[analyzer] Configuration file for scan-build.

[ayartsev]

Summary

A simple INI configuration file for scan-build with options - aliases to scan-build command line arguments. This configuration file can be used to customize scan-build runs as it is done via command line. Options from this config are processed before any command line arguments so equivalent command line arguments passed to scan-build will overwrite values obtained from config. An order in which options are passed to scan-build is the same in which they occur in the config.
PR8203 is related to the issue.
In the future I intend to use this config to also customize ccc/c++-analyzer script runs (to e.g pass special includes to Clang-analyzer, e.t.c).
Please review!

Currently looking through CMakeLists.txt in scan-build folder I do not exactly know where 'CommonStuff.pm' and 'scan-build.cfg' should be placed (BinFiles/LibexecFiles/ResourceFiles). Perhaps 'CommonStuff.pm' will go to the 'ResourceFiles' dir.
I am in favor of placing 'scan-build.cfg' to 'BinFiles' even bearing in mind hypothetical usage of this config by ccc/c++-analyzer scripts.

Tested the patch under Windows/Unix, any additional testing is appreciated!

[jroelofs]

CommonStuff.pm (which could stand to have a more descriptive name, libscanbuild.pm maybe?) and scan-build.cfg belong in share.

[jroelofs]

Also, how is this different from -analyzer-config?

[jroelofs]

Actually, now that I think about it more, scan-build.cfg doesn't belong in share at all. The location of an instance of it should be read in via a command-line argument, and it shouldn't be installed.

[ayartsev]

In D14629#288478, @jroelofs wrote:

CommonStuff.pm (which could stand to have a more descriptive name, libscanbuild.pm maybe?) and scan-build.cfg belong in share.

Got it, I'll rename CommonStuff.pm and put it to share in the next patch iteration.

In D14629#288485, @jroelofs wrote:

Also, how is this different from -analyzer-config?

-analyzer-config is used to transfers a number of options to the analyzer, while configuration file is used to customize scan-build, ccc-analyzer and c++-analyzer scripts.

In D14629#288495, @jroelofs wrote:

Actually, now that I think about it more, scan-build.cfg doesn't belong in share at all. The location of an instance of it should be read in via a command-line argument, and it shouldn't be installed.

I intend to add a command-line argument to customize the configuration file location, but there must be some default example configuration file to describe all available options and rules (scan-build.cfg from this patch).

Thanks for looking at this!

[zaks.anna]

Also, how is this different from -analyzer-config?

-analyzer-config is used to transfers a number of options to the analyzer, while configuration file is
used to customize scan-build, ccc-analyzer and c++-analyzer scripts.

Could you elaborate on this a bit and explain the motivation behind this feature.

Laszlo Nagy is working on "reimplementing scan-build in python" (and improving the tool) effort. If the configuration file is a desirable feature, we should make sure it is applicable to both scan-builds.

Also, we should strive to move away from using Perl.

Adding Laszlo as a reviewer.

[rizsotto.mailinglist]

I think the scan-build user experience would be improved by config file! And I really like how the clang-tidy guys were doing it. (In case if you are not familiar with it, copy from the help output)

Configuration files:
  clang-tidy attempts to read configuration for each source file from a
  .clang-tidy file located in the closest parent directory of the source
  file. If any configuration options have a corresponding command-line
  option, command-line option takes precedence. The effective
  configuration can be inspected using -dump-config:

  $ clang-tidy -dump-config - --
    ---
    Checks:          '-*,some-check'
    HeaderFilterRegex: ''
    AnalyzeTemporaryDtors: false
    User:            user
    CheckOptions:
      - key:             some-check.SomeOption
        value:           'some value'
    ...

So the major difference from the current patch would be:

• It's not a windows .ini file, but YAML/JSON syntax.
• No need to provide default config file and check into the source repository.
• Loading of the config file is a bit more complex logic, than taking value from a specific location.

Benefit for the user to have common usage pattern of the clang related analyzer tools. What do you think?

[jroelofs]

Are you aware of http://reviews.llvm.org/D9600 ?

[zaks.anna]

And I really like how the clang-tidy guys were doing it.

It would be great if the clang static analyzer config file would be similar to the clang-tidy one. We would avoid user confusion if they are consistent. Is some reuse possible? Currently, when clang-tidy calls the analyzer, can it pass the arguments to it through the config file?

[xazax.hun]

Clang tidy can pass per checker configuration to the static analyzer at the moment. I think it should not be hard to extend it.

[ayartsev]

Updated the patch following Laszlo's advises:
+ Changed config file format to YAML.
+ Dump current configuration.
+ Search for config file from the current folder and upper.

In D14629#294023, @jroelofs wrote:

Are you aware of http://reviews.llvm.org/D9600 ?

Learning Python and looking forward to the patch being committed.

Currently I try to launch scan-build over large codebases and note rooms for improvement and potentially helpful options. Find the configuration file that allows to customize ccc/c++-analyzer scripts very handy. The successful improvements to Perl implementation may be freely transferred to Python implementation in the future when the python patch gets in.

Tested the solution under Windows ant Unix.

[ayartsev]

Upd: Forgot to remove the printHash() subroutine from libscanbuild.pm. Using it locally for debug purpose.

[ayartsev]

Ping

[ayartsev]

Ping.

[zaks.anna]

Anton,

Have you investigated if we can reuse code from clang-tidy?

Also, the hope is that the python rewrite of scan-build will replace the current scan-build in the near future. Of cause, it still needs to be tested on Windows. I would really appreciate if you could help out with that since you are our most active contributor who uses Windows!

Unfortunately, because of this, I am not convinced that large improvements to the old scan-build script is where the effort should go at this point, especially, if the code cannot be reused by the new scan-build implementation.

tools/scan-build/CommonStuff.pm

+#!/usr/bin/env perl
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+##===----------------------------------------------------------------------===##
+#
+#  A set of routines common to scan-build and [ccc/c++]-analyzer scripts.
+#
+##===----------------------------------------------------------------------===##
+
+package CommonStuff;
+use strict;
+
+# Reads a section from INI (https://en.wikipedia.org/wiki/INI_file) file.
+# $INIFile       - path to the file to read options from.
+# $SectionToRead - name of the section to read options from.
+# $OptionsOut    - array reference to keep output, is being filled with
+#                  {Option =>, Value =>} elements as options are parsed.
+#                  Preserves an order in which the options occur in config file.
+sub ReadINI {
+  my ($INIFile, $SectionToRead, $OptionsOut) = @_;
+  my $CurrSection = "";
+  my $ErrMsg = undef;
+  my %OptionNames = (); # Just to check for duplicated options in a section.
+  my $SectionFound = 0;
+
+  return "$!" unless open(IN, "<", $INIFile);
+
+  while(<IN>) {
+    s/^\s+|\s+$//og;     # remove leading and trailing whitespaces
+    next if (/^$/o);    # skip empty lines
+    next if (/^[#;]/o); # skip comments
+
+    # Parse section name.
+    if (/^\[([a-zA-Z0-9\-_]+)\]$/o) {
+      $CurrSection = $1;
+      $SectionFound |= $CurrSection eq $SectionToRead;
+      next;
+    }
+    # Parse options.
+    elsif (/^([a-zA-Z0-9\-_]+)\s*=\s*(.*)$/o) {
+      next if $CurrSection ne $SectionToRead;
+      if (exists $OptionNames{$1}) {
+        $ErrMsg = "Duplicate option '$1' in section '[$CurrSection]'.";
+        last;
+      }
+      $OptionNames{$1} = 1;
+      # Fil output array with options with nonempty values.
+      push @$OptionsOut, { Option => $1, Value => $2 } if $2 ne "";
+    }
+    # Bad syntax.
+    else {
+      $ErrMsg = "Error parsing line '$_'.";
+      last;
+    }
+  }
+
+  $ErrMsg = "Section '[$SectionToRead]' not found."
+    unless $ErrMsg || $SectionFound;
+
+  close (IN);
+  return $ErrMsg;
+}
+
+1;

tools/scan-build/scan-build

 use File::Find;
 use File::Copy qw(copy);
 use File::Path qw( rmtree mkpath );
 use Term::ANSIColor;
 use Term::ANSIColor qw(:constants);
 use Cwd qw/ getcwd abs_path /;
 use Sys::Hostname;
 use Hash::Util qw(lock_keys);
+# Add scan-build dir to the list of places where perl looks for modules.
+use lib dirname(abs_path($0));
+use CommonStuff;
 
 my $Prog = "scan-build";
 my $BuildName;
 my $BuildDate;
 
 my $TERM = $ENV{'TERM'};
 my $UseColor = (defined $TERM and $TERM =~ 'xterm-.*color' and -t STDOUT
                 and defined $ENV{'SCAN_BUILD_COLOR'});
else {
         return "Cannot find an executable clang at '$Options{AnalyzerDiscoveryMethod}'\n";
       }
     }
   }
   return undef;
 }
 
 ##----------------------------------------------------------------------------##
+# Process options from config file.
+##----------------------------------------------------------------------------##
+
+my %CfgOptToCmdArg = (
+  ANALYZE_HEADERS_FLAG => "-analyze-headers",
+  OUTPUT_LOCATION => "-o",
+  KEEP_GOING_FLAG => "-k",
+  HTML_TITLE => "--html-title",
+  OUTPUT_FORMAT => undef,    # Alias for -plist/-plist-html. Possible values: [plist|plist-html].
+  STATUS_BUGS_FLAG => "--status-bugs",
+  USE_CC => "--use-cc",
+  USE_CPP => "--use-c++",
+  ANALYZER_TARGET => "--analyzer-target",
+  VERBOSITY_LEVEL => "-v",   # Possible values: [1|2|3].
+  VIEW_FLAG => "-V",
+  NO_FAILURE_REPORTS_FLAG => "-no-failure-reports",
+  STATS_FLAG => "-stats",
+  MAXLOOP => "-maxloop",
+  INTERNAL_STATS_FLAG => "-internal-stats",
+  USE_ANALYZER => "--use-analyzer",
+  KEEP_EMPTY_FLAG => "--keep-empty",
+  OVERRIDE_COMPILER_FLAG => "--override-compiler",
+  ANALYZER_CONFIG => "-analyzer-config",
+  ENABLE_CHECKERS => "-enable-checker",   # Value: checker names separated with commas.
+  DISABLE_CHECKERS => "-disable-checker", # Value: checker names separated with commas.
+  LOAD_PLUGINS => "-load-plugin"          # Value: plugin names separated with commas.
+);
+
+# Translates options obtained from config file to scan-build command line arguments.
+sub CfgOptionsToCmdArgs {
+  # $CfgOptions - reference to array of { Option, Value }.
+  # $CmdArgs - array for output.
+  my ($CfgOptions, $CmdArgs) = @_;
+
+  foreach my $OptionValuePair (@$CfgOptions) {
+    my $Option = $$OptionValuePair{Option};
+    my $Value = $$OptionValuePair{Value};
+
+    next if !defined $Value;
+
+    if (!exists $CfgOptToCmdArg{$Option} ) {
+      print "Warning: unrecognized option in config file: '$Option'.  Option ignored.";
+      next;
+    }
+    my $Arg = $CfgOptToCmdArg{$Option};
+
+    # OUTPUT_FORMAT
+    if ($Option eq "OUTPUT_FORMAT") {
+      if ($Value eq "plist") {
+        push @$CmdArgs, "-plist";
+      }
+      elsif ($Value eq "plist-html") {
+        push @$CmdArgs, "-plist-html";
+      } 
+      else {
+        print "Warning: '$Option' value must be [plist|plist-html]. Option ignored.\n";
+      }
+    }
+    # VERBOSITY_LEVEL
+    elsif ($Option eq "VERBOSITY_LEVEL") {
+      if ($Value !~ /\d/o || $Value < 1 || $Value > 3) {
+        print "Warning: '$Option' value must be [1|2|3]. Option ignored.\n";
+      }
+      push @$CmdArgs, $Arg if ($Value > 0);
+      push @$CmdArgs, $Arg if ($Value > 1);
+      push @$CmdArgs, $Arg if ($Value > 2);
+    }
+    # ENABLE_CHECKERS, DISABLE_CHECKERS, LOAD_PLUGINS
+    elsif ($Option eq "ENABLE_CHECKERS" ||
+           $Option eq "DISABLE_CHECKERS" ||
+           $Option eq "LOAD_PLUGINS") {
+      foreach (split /\s*,\s*/o, $Value) {
+        push @$CmdArgs, $Arg;
+        push @$CmdArgs, $_;
+      }
+    }
+    # Flags: {flag_name}_FLAG
+    elsif ($Option =~ /_FLAG$/o) {
+      if ($Value eq '1') {
+        push @$CmdArgs, $Arg;
+      }
+      elsif ($Value ne '0') {
+        print "Warning: '$Option' value must be [0|1]. Option ignored.\n";
+      }
+    }
+    # The rest options.
+    else {
+      push @$CmdArgs, $Arg;
+      push @$CmdArgs, $Value;
+    }
+  }
+}
+
+# Config file from scan-build dir.
+my $PathToCfg = dirname(abs_path($0))."/scan-build.cfg";
+# Array of { Option, Value }.
+my @OptionsFromCfg;
+# Options from config file translated to scan-build command line arguments.
+my @ArgsFromCfg;
+
+my $ErrorMsg = CommonStuff::ReadINI($PathToCfg, "scan-build", \@OptionsFromCfg);
+if (defined $ErrorMsg) {
+  print "Warning: error reading $PathToCfg: $ErrorMsg\n";
+}
+else {
+  CfgOptionsToCmdArgs(\@OptionsFromCfg, \@ArgsFromCfg);
+  # Add arguments from config file to the front of @ARGV.
+  unshift @ARGV, @ArgsFromCfg;
+}
+
+
+##----------------------------------------------------------------------------##
 # Process command-line arguments.
 ##----------------------------------------------------------------------------##
 
 my $RequestDisplayHelp = 0;
 my $ForceDisplayHelp = 0;
 
 sub ProcessArgs {
   my $Args = shift;

tools/scan-build/scan-build.cfg

+# A simple INI (https://en.wikipedia.org/wiki/INI_file) file.
+
+[scan-build] 
+# This section contains scan-build options which are aliases for scan-build
+# command-line arguments. This options are processed before any command line
+# arguments so equivalent command line arguments will overwrite values obtained
+# from this section.
+#
+# The order in which the options are passed to scan-build is the same in which
+# they occur in this section. You can freely change the options order or delete
+# options you don't like to see here.
+#
+# Options with no values are skipped.
+# {flag_name}_FLAG options can only take values 1 and 0.
+
+  # Alias for -analyze-headers.
+  ANALYZE_HEADERS_FLAG = 
+
+  # Alias for -o.
+  OUTPUT_LOCATION = 
+
+  # Alias for -k (--keep-going).
+  KEEP_GOING_FLAG = 
+
+  # Alias for --html-title.
+  HTML_TITLE = 
+
+  # Alias for -plist/-plist-html.
+  # Possible values: [plist|plist-html].
+  OUTPUT_FORMAT = 
+
+  # Alias for --status-bugs.
+  STATUS_BUGS_FLAG = 
+
+  # Alias for --use-cc.
+  USE_CC = 
+
+  # Alias for --use-c++.
+  USE_CPP = 
+
+  # Alias for --analyzer-target.
+  ANALYZER_TARGET = 
+
+  # Alias for -v.
+  #Possible values: [1|2|3].
+  VERBOSITY_LEVEL = 
+
+  # Alias for -V (--view).
+  VIEW_FLAG = 
+
+  # Alias for -no-failure-reports.
+  NO_FAILURE_REPORTS_FLAG = 
+
+  # Alias for -stats.
+  STATS_FLAG = 
+
+  # Alias for -maxloop.
+  MAXLOOP = 
+
+  # Alias for -internal-stats.
+  INTERNAL_STATS_FLAG = 
+
+  # Alias for --use-analyzer.
+  USE_ANALYZER = 
+
+  # Alias for --keep-empty.
+  KEEP_EMPTY_FLAG = 
+
+  # Alias for --override-compiler.
+  OVERRIDE_COMPILER_FLAG = 
+
+  # Alias for -analyzer-config.
+  ANALYZER_CONFIG = 
+
+  # Alias for -enable-checker.
+  # Value: checker names separated by commas.
+  ENABLE_CHECKERS = 
+
+  # Alias for -disable-checker.
+  # Value: checker names separated by commas.
+  DISABLE_CHECKERS = 
+
+  # Alias for -load-plugin.
+  # Value: plugin names separated by commas.
+  LOAD_PLUGINS =