Index: tools/codechecker/README.md =================================================================== --- /dev/null +++ tools/codechecker/README.md @@ -0,0 +1,131 @@ + +----- +# Introduction +CodeChecker is a static analysis infrastructure built on [Clang Static Analyzer](http://clang-analyzer.llvm.org/). + +CodeChecker replaces [scan-build](http://clang-analyzer.llvm.org/scan-build.html) in Clang Static Analyzer in Linux systems. + +Main features: + * store the result of multiple large analysis run results efficiently + * run multiple analyzers, currently Clang Static Analyzer and Clang-Tidy is supported + * dynamic web based defect viewer + * a PostgreSQL/SQLite based defect storage & management (both are optional, results can be shown on standard output in quickcheck mode) + * update analyzer results only for modified files (depends on the build system) + * compare analysis results (new/resolved/unresolved bugs compared to a baseline) + * filter analysis results (checker name, severity, source file name ...) + * skip analysis in specific source directories if required + * suppression of false positives (in config file or in the source) + * Thrift API based server-client model for storing bugs and viewing results. + * It is possible to connect multiple bug viewers. Currently a web-based viewer and a command line viewer are provided. + +You can find a high level overview about the infrastructure in the presentation +at the [2015 Euro LLVM](http://llvm.org/devmtg/2015-04/) Conference: + +__Industrial Experiences with the Clang Static Analysis Toolset +_Daniel Krupp, Gyorgy Orban, Gabor Horvath and Bence Babati___ ([ Slides](http://llvm.org/devmtg/2015-04/slides/Clang_static_analysis_toolset_final.pdf)) + +## Important Limitations +CodeChecker requires some new features from clang to work properly. +If your clang version does not have these features you will see in debug log the following messages: + + * `Check name wasn't found in the plist file.` --> use clang = 3.7 or trunk@r228624; otherwise CodeChecker makes a guess based on the report message + * `Hash value wasn't found in the plist file.` --> update for a newer clang version; otherwise CodeChecker generates a simple hash based on the filename and the line content, this method is applied for Clang Tidy results too, because Clang Tidy does not support bug identifier hash generation currently + +## Linux +For a more detailed dependency list see [Requirements](docs/deps.md) +### Basic dependecy install & setup +Tested on Ubuntu LTS 14.04.2 +~~~~~~{.sh} + +# get ubuntu packages +# clang-3.6 can be replaced by any later versions of clang +sudo apt-get install clang-3.6 doxygen build-essential thrift-compiler python-virtualenv gcc-multilib git wget + +# create new python virtualenv +virtualenv -p /usr/bin/python2.7 ~/checker_env +# activate virtualenv +source ~/checker_env/bin/activate + +# get source code +git clone https://github.com/Ericsson/codechecker.git +cd codechecker + +# install required basic python modules +pip install -r .ci/basic_python_requirements + +# create codechecker package +./build_package.py -o ~/codechecker_package +cd .. +~~~~~~ + + +### Check a test project +#### Check if clang or clang tidy is available +~~~~~~{.sh} +which clang +which clang-tidy +~~~~~~ +If 'clang' or 'clang-tidy' commands are not available the package can be configured to use another/newer clang binary for the analisys. +Edit the 'CodeChecker/config/package_layout.json' config files "runtime/analyzers" +section in the generated package and modify the analyzers section to the analyzers +available in the PATH +``` +"analyzers" : { + "clangsa" : "clang-3.6", + "clang-tidy" : "clang-tidy-3.6" + }, +``` + +#### Activate virtualenv +~~~~~~{.sh} +source ~/checker_env/bin/activate +~~~~~~ + +#### Add package bin directory to PATH. +This step can be skipped if you always give the path of CodeChecker command. +~~~~~~{.sh} +export PATH=~/codechecker_package/CodeChecker/bin:$PATH +~~~~~~ + +#### Check the project +Check the project using SQLite. The database is placed in the working +directory which can be provided by -w flag (~/.codechecker by default). +~~~~~~{.sh} +CodeChecker check -n test_project_check -b "cd my_test_project && make clean && make" +~~~~~~ + +#### Start web server to view the results +~~~~~~{.sh} +CodeChecker server +~~~~~~ + +#### View the results with firefox +~~~~~~{.sh} +firefox http://localhost:8001 +~~~~~~ + +See user guide for further configuration and check options. + +## Additional documentations + +[User guide](docs/user_guide.md) + +[Use with PostgreSQL database](docs/postgresql_setup.md) + +[Command line usage_examples](docs/usage.md) + +[Checker documentation](docs/checker_docs.md) + +[Architecture overview](docs/architecture.md) + +[Requirements](docs/deps.md) + +[Package layout](docs/package_layout.md) + +[Thrift api](thrift_api/thrift_api.md) + +[External source dependencies](docs/deps.md) + +[Test documentation](tests/functional/package_test.md) + +[Database schema migration](docs/db_schema_guide.md) Index: tools/codechecker/alembic.ini =================================================================== --- /dev/null +++ tools/codechecker/alembic.ini @@ -0,0 +1,67 @@ +# A generic, single database configuration. + +[alembic] +# path to migration scripts +script_location = libcodechecker/db_migrate + +# template used to generate migration files +# file_template = %%(rev)s_%%(slug)s + +# max length of characters to apply to the +# "slug" field +#truncate_slug_length = 40 + +# set to 'true' to run the environment during +# the 'revision' command, regardless of autogenerate +# revision_environment = false + +# set to 'true' to allow .pyc and .pyo files without +# a source .py file to be detected as revisions in the +# versions/ directory +# sourceless = false + +# version location specification; this defaults +# to database/versions. When using multiple version +# directories, initial revisions must be specified with --version-path +# version_locations = %(here)s/bar %(here)s/bat database/versions + +# the output encoding used when revision files +# are written from script.py.mako +# output_encoding = utf-8 + +sqlalchemy.url = postgres://codechecker@localhost:5432/codechecker + +# Logging configuration +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S Index: tools/codechecker/bin/CodeChecker =================================================================== --- /dev/null +++ tools/codechecker/bin/CodeChecker @@ -0,0 +1,106 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +''' + used to kickstart codechecker + save original environment without modifications + used to run the logging in the same env +''' + +import os +import sys +import signal +import subprocess +import pickle +import ntpath +import tempfile +import shutil + +procPool = [] + +def run_codechecker(original_env, checker_env): + ''' + run the codechecker + original_env - will be saved for later usage by the build action logger + checker_env - codechecker will be run in the checker env + ''' + + package_bin = os.path.dirname(os.path.realpath(__file__)) + package_root, bin_dir = ntpath.split(package_bin) + + python = os.path.join('python') + common_lib = os.path.join(package_root, + 'cc_lib', + 'python2.7') + + gen_lib = os.path.join(package_root, + 'cc_lib', + 'python2.7', + 'codechecker_gen') + + checker_env['PYTHONPATH'] = common_lib + ':' + gen_lib + + checker_env['CC_PACKAGE_ROOT'] = \ + os.path.realpath(os.path.join(package_bin, os.pardir)) + + codechecker_main = os.path.join(package_root, + 'cc_bin', + 'CodeChecker.py') + + checker_cmd = [] + checker_cmd.append(python) + checker_cmd.append(codechecker_main) + checker_cmd.extend(sys.argv[1:]) + + proc = subprocess.Popen(checker_cmd, env=checker_env) + procPool.append(proc.pid) + + proc.wait() + sys.exit(proc.returncode) + +def main(): + + original_env = os.environ.copy() + checker_env = original_env + + tmp_dir = tempfile.mkdtemp() + + original_env_file = os.path.join(tmp_dir, 'original_env.pickle') + + try: + with open(original_env_file, 'wb') as env_save: + pickle.dump(original_env, env_save) + + checker_env['CODECHECKER_ORIGINAL_BUILD_ENV'] = original_env_file + except Exception as ex: + print('Saving original build environment failed') + print(ex) + + def signal_term_handler(sig, frame): + pid = os.getpid() + for p in procPool: + os.kill(p, signal.SIGINT) + + # remove temporary directory + try: + shutil.rmtree(tmp_dir) + except Exception as ex: + print('Failed to remove temporary directory: ' + tmp_dir) + print('Manual cleanup is required.') + print(ex) + + sys.exit(1) + + signal.signal(signal.SIGTERM, signal_term_handler) + signal.signal(signal.SIGINT, signal_term_handler) + + run_codechecker(original_env, checker_env) + + shutil.rmtree(tmp_dir) + +if __name__ == "__main__": + main() Index: tools/codechecker/bin/CodeChecker.py =================================================================== --- /dev/null +++ tools/codechecker/bin/CodeChecker.py @@ -0,0 +1,401 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +''' +main codechecker script + +''' +import os +import sys +import signal +import argparse + +import shared + +from libcodechecker import logger +from libcodechecker import arg_handler +from libcodechecker import util +from libcodechecker.analyzers import analyzer_types +from libcodechecker.cmdline_client import cmd_line_client + + +LOG = logger.get_new_logger('MAIN') + +analyzers = ' '.join(list(analyzer_types.supported_analyzers)) + + +class OrderedCheckersAction(argparse.Action): + ''' + Action to store enabled and disabled checkers + and keep ordering from command line + + Create separate lists based on the checker names for + each analyzer + ''' + def __init__(self, option_strings, dest, nargs=None, **kwargs): + if nargs is not None: + raise ValueError("nargs not allowed") + super(OrderedCheckersAction, self).__init__(option_strings, dest, **kwargs) + + def __call__(self, parser, namespace, value, option_string=None): + + if 'ordered_checkers' not in namespace: + setattr(namespace, 'ordered_checkers', []) + ordered_checkers = namespace.ordered_checkers + if self.dest == 'enable': + ordered_checkers.append((value, True)) + else: + ordered_checkers.append((value, False)) + + setattr(namespace, 'ordered_checkers', ordered_checkers) + + +# ------------------------------------------------------------------------------ +class DeprecatedOptionAction(argparse.Action): + ''' + Deprecated argument action + ''' + def __init__(self, + option_strings, + dest, + nargs=None, + const=None, + default=None, + type=None, + choices=None, + required=False, + help=None, + metavar=None): + super(DeprecatedOptionAction, self).__init__(option_strings, + dest, + nargs='?', + const='deprecated_option', + default=None, + type=None, + choices=None, + required=False, + help='DEPRECATED argument!', + metavar='DEPRECATED') + + def __call__(self, parser, namespace, value=None, option_string=None): + LOG.warning("Deprecated command line option in use: '" + option_string + "'") + + +def add_database_arguments(parser): + '''Helper method for adding database arguments to an argument parser.''' + + parser.add_argument('--sqlite', action=DeprecatedOptionAction) + parser.add_argument('--postgresql', dest="postgresql", + action='store_true', required=False, + help='Use PostgreSQL database.') + parser.add_argument('--dbport', type=int, dest="dbport", + default=5432, required=False, + help='Postgres server port.') + # WARNING dbaddress default value influences workspace creation (SQLite) + parser.add_argument('--dbaddress', type=str, dest="dbaddress", + default="localhost", required=False, + help='Postgres database server address') + parser.add_argument('--dbname', type=str, dest="dbname", + default="codechecker", required=False, + help='Name of the database.') + parser.add_argument('--dbusername', type=str, dest="dbusername", + default='codechecker', required=False, + help='Database user name.') + + +def add_analyzer_arguments(parser): + """ + analyzer related arguments + """ + parser.add_argument('-e', '--enable', + default=argparse.SUPPRESS, + action=OrderedCheckersAction, + help='Enable checker.') + parser.add_argument('-d', '--disable', + default=argparse.SUPPRESS, + action=OrderedCheckersAction, + help='Disable checker.') + parser.add_argument('--keep-tmp', action="store_true", + dest="keep_tmp", required=False, + help='''\ +Keep temporary report files generated during the analysis.''') + + parser.add_argument('--analyzers', nargs='+', + dest="analyzers", required=False, + default=[analyzer_types.CLANG_SA, analyzer_types.CLANG_TIDY], + help="""Select which analyzer should be enabled.\nCurrently supported analyzers are: """ + analyzers + """\ne.g. '--analyzers """ + analyzers +"'") + + parser.add_argument('--saargs', dest="clangsa_args_cfg_file", + required=False, default=argparse.SUPPRESS, + help='''\ +File with arguments which will be forwarded directly to the Clang static analyzer without modifiaction''') + + parser.add_argument('--tidyargs', dest="tidy_args_cfg_file", + required=False, default=argparse.SUPPRESS, + help='''\ +File with arguments which will be forwarded directly to the Clang tidy analyzer without modifiaction''') + +# ------------------------------------------------------------------------------ +def main(): + ''' + codechecker main command line + ''' + + def signal_handler(sig, frame): + ''' + Without this handler the postgreSql + server does not terminated at signal + ''' + sys.exit(1) + + signal.signal(signal.SIGINT, signal_handler) + signal.signal(signal.SIGTERM, signal_handler) + + try: + parser = argparse.ArgumentParser( + prog='CodeChecker', + formatter_class=argparse.RawDescriptionHelpFormatter, + description=''' +Run the CodeChecker source analyzer framework. +See the subcommands for specific features.''', + epilog=''' +Example usage: +-------------- +Analyzing a project with default settings: +CodeChecker check -w ~/workspace -b "cd ~/myproject && make" -n myproject + +Start the viewer to see the results: +CodeChecker server -w ~/workspace + +See the results in a web browser: localhost:8001 +See results in the command line: CodeChecker cmd results -p 8001 -n myproject + +To analyze a small project quickcheck feature can be used. +The results will be printed only to the standard output. (No database will be used) + +CodeChecker quickcheck -w ~/workspace -b "cd ~/myproject && make" +''') + + subparsers = parser.add_subparsers(help='commands') + + workspace_help_msg = """Directory where the codechecker can store analysis related data.""" + + name_help_msg = """Name of the analysis.""" + + jobs_help_msg = """Number of jobs. Start multiple processes for faster analisys"""; + + log_argument_help_msg="""Path to the log file which is created during the build. \nIf there is an already generated log file with the compilation commands\ngenerated by 'CodeChecker log' or 'cmake -DCMAKE_EXPORT_COMPILE_COMMANDS' \nCodechecker check can use it for the analisys in that case running the original build will \nbe left out from the analysis process (no log is needed).""" + + # -------------------------------------- + # check commands + check_parser = subparsers.add_parser('check', + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + help='''\ +Run the supported source code analyzers on a project''') + check_parser.add_argument('-w', '--workspace', type=str, + default=util.get_default_workspace(), + dest="workspace", + help=workspace_help_msg) + check_parser.add_argument('-n', '--name', type=str, + dest="name", required=True, + default=argparse.SUPPRESS, + help=name_help_msg) + checkgroup = check_parser.add_mutually_exclusive_group(required=True) + checkgroup.add_argument('-b', '--build', type=str, dest="command", + default=argparse.SUPPRESS, + required=False, help='''\ +Build command which is used to build the project''') + checkgroup.add_argument('-l', '--log', type=str, dest="logfile", + default=argparse.SUPPRESS, + required=False, + help=log_argument_help_msg) + check_parser.add_argument('-j', '--jobs', type=int, dest="jobs", + default=1, required=False, + help=jobs_help_msg) + check_parser.add_argument('-u', '--suppress', type=str, dest="suppress", + default=argparse.SUPPRESS, + required=False, help="""Path to suppress file. \nSuppress file can be used to suppress analysis results during the analisys.\nIt is based on the bug identifier generated by the compiler which is experimental.\nDo not depend too much on this file because identifier or file format can be changed.\nFor other in source suppress features see the user guide.""") + check_parser.add_argument('-c', '--clean', + default=argparse.SUPPRESS, + action=DeprecatedOptionAction) + check_parser.add_argument('--update', action=DeprecatedOptionAction, + dest="update", default=False, required=False, + help='''\ +Incremental parsing, update the results of a previous run. +Only the files changed since the last build will be reanalyzed. Depends on the build system.''') + + check_parser.add_argument('--force', action="store_true", + dest="force", default=False, required=False, + help='''\ +Delete analisys results form the database if a run with the given name already exists''') + check_parser.add_argument('-s', '--skip', type=str, dest="skipfile", + default=argparse.SUPPRESS, + required=False, help='Path to skip file.') + add_analyzer_arguments(check_parser) + add_database_arguments(check_parser) + check_parser.set_defaults(func=arg_handler.handle_check) + + # -------------------------------------- + # quickcheck commands + qcheck_parser = subparsers.add_parser('quickcheck', + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + help='''\ +Run CodeChecker for a project without database.''') + qcheckgroup = qcheck_parser.add_mutually_exclusive_group(required=True) + qcheckgroup.add_argument('-b', '--build', type=str, dest="command", + default=argparse.SUPPRESS, + required=False, help='Build command.') + qcheckgroup.add_argument('-l', '--log', type=str, dest="logfile", + required=False, + default=argparse.SUPPRESS, + help=log_argument_help_msg) + qcheck_parser.add_argument('-s', '--steps', action="store_true", + dest="print_steps", help='Print steps.') + add_analyzer_arguments(qcheck_parser) + qcheck_parser.set_defaults(func=arg_handler.handle_quickcheck) + + + # -------------------------------------- + # log commands + logging_parser = subparsers.add_parser('log', + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + help='''\ +Runs the given build command. During the build the compilation commands are collected and +stored into a compilation command json file (no analisys is done during the build).''') + logging_parser.add_argument('-o', '--output', type=str, dest="logfile", + default=argparse.SUPPRESS, + required=True, help='Path to the log file.') + logging_parser.add_argument('-b', '--build', type=str, dest="command", + default=argparse.SUPPRESS, + required=True, help='Build command.') + logging_parser.set_defaults(func=arg_handler.handle_log) + + # -------------------------------------- + # checkers parser + checkers_parser = subparsers.add_parser('checkers', + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + help="""List the available checkers for the supported analyzers and show their default status (+ for being enabled, - for being disabled by default).""") + checkers_parser.add_argument('--analyzers', nargs='+', + dest="analyzers", required=False, + help="""Select which analyzer checkers should be listed.\nCurrently supported analyzers:\n""" + analyzers) + checkers_parser.set_defaults(func=arg_handler.handle_list_checkers) + + # -------------------------------------- + # server + server_parser = subparsers.add_parser('server', + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + help='Start the codechecker web server.') + server_parser.add_argument('-w', '--workspace', type=str, + dest="workspace", + default=util.get_default_workspace(), + help=workspace_help_msg) + server_parser.add_argument('-v', '--view-port', type=int, dest="view_port", + default=8001, required=False, + help='Port used for viewing.') + server_parser.add_argument('-u', '--suppress', type=str, dest="suppress", + required=False, help='Path to suppress file.') + server_parser.add_argument('--not-host-only', action="store_true", + dest="not_host_only", + help='Viewing the results is possible not \ + only by browsers or clients started locally.') + server_parser.add_argument('--check-port', type=int, dest="check_port", + default=None, required=False, + help='Port used for checking.') + server_parser.add_argument('--check-address', type=str, + dest="check_address", default="localhost", + required=False, help='Server address.') + add_database_arguments(server_parser) + server_parser.set_defaults(func=arg_handler.handle_server) + + # -------------------------------------- + # cmd_line + cmd_line_parser = subparsers.add_parser('cmd', + help='Command line client') + cmd_line_client.register_client_command_line(cmd_line_parser) + + # -------------------------------------- + # debug parser + debug_parser = subparsers.add_parser('debug', + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + help="""Generate gdb debug dump files for all the failed compilation commands in the last analyzer run.\nRequires a database with the failed compilation commands""") + debug_parser.add_argument('-w', '--workspace', type=str, + dest="workspace", + default=util.get_default_workspace(), + help=workspace_help_msg) + debug_parser.add_argument('-f', '--force', action="store_true", + dest="force", required=False, default=False, + help='Overwrite already generated files.') + add_database_arguments(debug_parser) + debug_parser.set_defaults(func=arg_handler.handle_debug) + + # -------------------------------------- + # plist parser + plist_parser = subparsers.add_parser('plist', + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + help='Parse plist files in the given directory.') + plist_parser.add_argument('-w', '--workspace', type=str, + dest="workspace", + default=util.get_default_workspace(), + help=workspace_help_msg) + plist_parser.add_argument('-n', '--name', type=str, + dest="name", required=True, + default=argparse.SUPPRESS, + help=name_help_msg) + plist_parser.add_argument('-d', '--directory', type=str, + dest="directory", required=True, + help='Path of a directory containing plist files to parse.') + plist_parser.add_argument('-j', '--jobs', type=int, dest="jobs", + default=1, required=False, + help=jobs_help_msg) + plist_parser.add_argument('-s', '--steps', action="store_true", + dest="print_steps", help='Print steps.') + plist_parser.add_argument('--stdout', action="store_true", dest="stdout", + required=False, default=False, + help="Print results to stdout instead of database.") + plist_parser.add_argument('--force', action="store_true", + dest="force", default=False, required=False, + help='''\ +Delete analisys results form the database if a run with the given name already exists''') + add_database_arguments(plist_parser) + plist_parser.set_defaults(func=arg_handler.handle_plist) + + # -------------------------------------- + # package version info + version_parser = subparsers.add_parser('version', + help='Print package version information.') + version_parser.set_defaults(func=arg_handler.handle_version_info) + + args = parser.parse_args() + args.func(args) + + except KeyboardInterrupt as kb_err: + LOG.info(str(kb_err)) + LOG.info("Interupted by user...") + sys.exit(1) + + except shared.ttypes.RequestFailed as thrift_ex: + LOG.info("Server error.") + LOG.info("Error code: " + str(thrift_ex.error_code)) + LOG.info("Error message: " + str(thrift_ex.message)) + sys.exit(1) + + # Handle all exception, but print stacktrace. It is needed for atexit. + # atexit does not work correctly when an unhandled exception occured. + # So in this case, the servers left running when the script exited. + except Exception: + import traceback + traceback.print_exc(file=sys.stdout) + sys.exit(2) + + +# ------------------------------------------------------------------------------ +if __name__ == "__main__": + LOG.debug(sys.path) + LOG.debug(sys.version) + LOG.debug(sys.executable) + LOG.debug(os.environ.get('LD_LIBRARY_PATH')) + + main() Index: tools/codechecker/config/checker_severity_map.json =================================================================== --- /dev/null +++ tools/codechecker/config/checker_severity_map.json @@ -0,0 +1,123 @@ +{ + "security.FloatLoopCounter": "MEDIUM", + "security.insecureAPI.UncheckedReturn": "MEDIUM", + "security.insecureAPI.getpw": "MEDIUM", + "security.insecureAPI.gets": "MEDIUM", + "security.insecureAPI.mkstemp": "MEDIUM", + "security.insecureAPI.mktemp": "MEDIUM", + "security.insecureAPI.rand": "MEDIUM", + "security.insecureAPI.strcpy": "MEDIUM", + "security.insecureAPI.vfork": "MEDIUM", + "unix.API": "MEDIUM", + "unix.Malloc": "MEDIUM", + "unix.MallocSizeof": "MEDIUM", + "unix.MismatchedDeallocator": "MEDIUM", + "unix.Vfork": "MEDIUM", + "unix.cstring.BadSizeArg": "MEDIUM", + "unix.cstring.NullArg": "MEDIUM", + "nullability.NullPassedToNonnull": "HIGH", + "nullability.NullReturnedFromNonnull": "HIGH", + "nullability.NullableDereferenced": "MEDIUM", + "nullability.NullablePassedToNonnull": "MEDIUM", + "nullability.NullablePassedToNonnull": "MEDIUM", + "core.CallAndMessage": "HIGH", + "core.DivideZero": "HIGH", + "core.DynamicTypePropagation": "MEDIUM", + "core.NonNullParamChecker": "HIGH", + "core.NullDereference": "HIGH", + "core.StackAddressEscape": "HIGH", + "core.UndefinedBinaryOperatorResult": "MEDIUM", + "core.VLASize": "MEDIUM", + "core.builtin.BuiltinFunctions": "MEDIUM", + "core.builtin.NoReturnFunctions": "MEDIUM", + "core.uninitialized.ArraySubscript": "MEDIUM", + "core.uninitialized.Assign": "MEDIUM", + "core.uninitialized.Branch": "MEDIUM", + "core.uninitialized.CapturedBlockVariable": "MEDIUM", + "core.uninitialized.UndefReturn": "HIGH", + "cplusplus.NewDelete": "HIGH", + "cplusplus.NewDeleteLeaks": "HIGH", + "deadcode.DeadStores": "LOW", + "llvm.Conventions": "LOW", + "cert-dcl03-c": "MEDIUM", + "cert-dcl50-cpp": "LOW", + "cert-dcl54-cpp": "MEDIUM", + "cert-dcl59-cpp": "MEDIUM", + "cert-err52-cpp": "LOW", + "cert-err60-cpp": "MEDIUM", + "cert-err61-cpp": "MEDIUM", + "cert-fio38-c": "HIGH", + "cert-oop11-cpp": "MEDIUM", + "cppcoreguidelines-c-copy-assignment-signature": "MEDIUM", + "cppcoreguidelines-pro-bounds-array-to-pointer-decay": "LOW", + "cppcoreguidelines-pro-bounds-pointer-arithmetic": "LOW", + "cppcoreguidelines-pro-type-const-cast": "LOW", + "cppcoreguidelines-pro-type-cstyle-cast": "LOW", + "cppcoreguidelines-pro-type-reinterpret-cast": "LOW", + "cppcoreguidelines-pro-type-static-cast-downcast": "LOW", + "cppcoreguidelines-pro-type-union-access": "LOW", + "cppcoreguidelines-pro-type-vararg": "LOW", + "google-build-explicit-make-pair": "MEDIUM", + "google-build-namespaces": "MEDIUM", + "google-build-using-namespace": "LOW", + "google-explicit-constructor": "MEDIUM", + "google-global-names-in-headers": "HIGH", + "google-readability-braces-around-statements": "LOW", + "google-readability-casting": "LOW", + "google-readability-function-size": "LOW", + "google-readability-namespace-comments": "LOW", + "google-readability-redundant-smartptr-get": "MEDIUM", + "google-readability-todo": "LOW", + "google-runtime-int": "LOW", + "google-runtime-member-string-references": "LOW", + "google-runtime-memset": "HIGH", + "google-runtime-operator": "MEDIUM", + "llvm-header-guard": "LOW", + "llvm-include-order": "LOW", + "llvm-namespace-comment": "LOW", + "llvm-twine-local": "LOW", + "misc-argument-comment": "LOW", + "misc-assert-side-effect": "MEDIUM", + "misc-assign-operator-signature": "MEDIUM", + "misc-bool-pointer-implicit-conversion": "LOW", + "misc-inaccurate-erase": "HIGH", + "misc-inefficient-algorithm": "MEDIUM", + "misc-macro-parentheses": "MEDIUM", + "misc-macro-repeated-side-effects": "MEDIUM", + "misc-move-const-arg": "MEDIUM", + "misc-move-constructor-init": "MEDIUM", + "misc-new-delete-overloads": "MEDIUM", + "misc-noexcept-move-constructor": "MEDIUM", + "misc-non-copyable-objects": "HIGH", + "misc-sizeof-container": "HIGH", + "misc-static-assert": "MEDIUM", + "misc-swapped-arguments": "HIGH", + "misc-throw-by-value-catch-by-reference": "MEDIUM", + "misc-undelegated-constructor": "MEDIUM", + "misc-uniqueptr-reset-release": "MEDIUM", + "misc-unused-alias-decls": "LOW", + "misc-unused-parameters": "LOW", + "misc-unused-raii": "HIGH", + "modernize-loop-convert": "LOW", + "modernize-make-unique": "LOW", + "modernize-pass-by-value": "LOW", + "modernize-redundant-void-arg": "LOW", + "modernize-replace-auto-ptr": "LOW", + "modernize-shrink-to-fit": "LOW", + "modernize-use-auto": "LOW", + "modernize-use-default": "LOW", + "modernize-use-nullptr": "LOW", + "modernize-use-override": "LOW", + "readability-braces-around-statements": "LOW", + "readability-container-size-empty": "LOW", + "readability-else-after-return": "LOW", + "readability-function-size": "LOW", + "readability-identifier-naming": "LOW", + "readability-implicit-bool-cast": "LOW", + "readability-inconsistent-declaration-parameter-name": "LOW", + "readability-named-parameter": "LOW", + "readability-redundant-smartptr-get": "LOW", + "readability-redundant-string-cstr": "LOW", + "readability-simplify-boolean-expr": "MEDIUM", + "readability-uniqueptr-delete-release": "LOW" +} Index: tools/codechecker/config/config.json =================================================================== --- /dev/null +++ tools/codechecker/config/config.json @@ -0,0 +1,46 @@ +{ + "environment_variables": { + "env_package_root" : "CC_PACKAGE_ROOT", + "env_verbose_name" : "CC_VERBOSE_LEVEL", + "env_alchemy_verbose_name" : "CC_ALCHEMY_LOG_LEVEL", + "env_path" : "PATH", + "env_ld_lib_path" : "LD_LIBRARY_PATH", + "cc_logger_bin" : "CC_LOGGER_BIN", + "cc_logger_file" : "CC_LOGGER_FILE", + "cc_logger_compiles" : "CC_LOGGER_GCC_LIKE", + "ld_preload" : "LD_PRELOAD", + "codechecker_enable_check" : "CODECHECKER_ENABLE_CHECK", + "codechecker_disable_check" : "CODECHECKER_DISABLE_CHECK", + "codechecker_workspace" : "CODECHECKER_WORKSPACE" + }, + "package_variables": { + "default_db_username" : "codechecker", + "pgsql_data_dir_name" : "pgsql_data", + "path_dumps_name" : "dumps" + }, + "checker_config": { + "clangsa_checkers" : [ + {"core" : true }, + {"unix" : true }, + {"deadcode" : true }, + {"cplusplus" : true }, + {"security.insecureAPI.UncheckedReturn" : true }, + {"security.insecureAPI.getpw" : true }, + {"security.insecureAPI.gets" : true }, + {"security.insecureAPI.mktemp" : true }, + {"security.insecureAPI.mkstemp" : true }, + {"security.insecureAPI.vfork" : true } + ], + "clang-tidy_checkers" : [ + {"clang-diagnostic-" : true }, + {"cert-fio38-c" : true }, + {"google-global-names-in-headers" : true }, + {"google-runtime-memset" : true }, + {"misc-inaccurate-erase" : true }, + {"misc-non-copyable-objects" : true }, + {"misc-sizeof-container" : true }, + {"misc-swapped-arguments" : true }, + {"misc-unused-raii" : true } + ] + } +} Index: tools/codechecker/config/config.md =================================================================== --- /dev/null +++ tools/codechecker/config/config.md @@ -0,0 +1,22 @@ + +# Package configuration + +### Checker severity map +checker_severity_map.json file contains a mapping between a +checker name and a severity level. Severity levels can be found in the shared.thrift file. + +### Package configuration + * environment variables section + Contains enviroment variable names set and used during the static analysis + * package variables section + Default database username which will be used to initialize postgres database. + * checker config section + + checkers + This section contains the default checkers set used for analysis. + The order of the checkers will be kept. (To enable set to true, to disable set to false) + +### gdb script +Contains an automated gdb script which can be used for debug. In debug mode the failed build commands will be rerun with gdb. + +### version +Version file contains version information for the package Index: tools/codechecker/config/gdbScript.gdb =================================================================== --- /dev/null +++ tools/codechecker/config/gdbScript.gdb @@ -0,0 +1,13 @@ +set follow-fork-mode child +set print pretty on +set print array on +set print demangle on +run +where +backtrace full +frame +info args +info locals +show print demangle +list +quit Index: tools/codechecker/config/package_layout.json =================================================================== --- /dev/null +++ tools/codechecker/config/package_layout.json @@ -0,0 +1,41 @@ +{ + "static" : { + "bin": "bin", + "cc_bin": "cc_bin", + "lib": "lib", + "plugin": "plugin", + "www": "www", + "docs": "www/docs", + "checker_md_docs": "www/docs/checker_md_docs", + "web_client": "www/scripts/codechecker-api", + "web_client_plugins": "www/scripts/plugins", + "web_client_codemirror": "www/scripts/plugins/codemirror", + "web_client_dojo": "www/scripts/plugins/dojo", + "web_client_highlightjs": "www/scripts/plugins/highlightjs", + "web_client_jsplumb": "www/scripts/plugins/jsplumb", + "web_client_marked": "www/scripts/plugins/marked", + "js_thrift": "www/scripts/plugins/thrift", + "config": "config", + "ld_logger": "ld_logger", + "codechecker_modules": "cc_lib/python2.7", + "libcodechecker": "cc_lib/python2.7/libcodechecker", + "codechecker_gen": "cc_lib/python2.7/codechecker_gen", + "codechecker_db_model": "cc_lib/python2.7/db_model", + "codechecker_db_migrate": "cc_lib/python2.7/db_migrate", + "storage_server_modules": "cc_lib/python2.7/storage_server", + "viewer_server_modules": "cc_lib/python2.7/viewer_server", + "cmdline_client": "cc_lib/python2.7/cmdline_client" + }, + "runtime" : { + "analyzers" : { + "clangsa" : "clang", + "clang-tidy" : "clang-tidy" + }, + "ld_logger_bin" : "bin/ldlogger", + "ld_logger_lib_name" : "ldlogger.so", + "ld_logger_lib_path" : "ld_logger/lib", + "version_file" : "config/version.json", + "checkers_severity_map_file" : "config/checker_severity_map.json", + "gdb_config_file" : "config/gdbScript.gdb" + } +} Index: tools/codechecker/config/version.json =================================================================== --- /dev/null +++ tools/codechecker/config/version.json @@ -0,0 +1,10 @@ +{ + "version":{ + "major" : "5", + "minor" : "4" + }, + "db_version":{ + "major" : "5", + "minor" : "0" + } +} Index: tools/codechecker/docs/architecture.md =================================================================== --- /dev/null +++ tools/codechecker/docs/architecture.md @@ -0,0 +1,79 @@ + +# Architecture overview +``` +Architecture + + .---------. + | Web | + .-------------. .----------. | browser | + | Buildaction | ---> .----------. | client | + | job list | ---> .----------.------. '---------'.---------. + | | ---> | Analyzer |------| ^ | Command | + '-------------' | client |------| |---->| line | + ^ '----------' | | | client | + | | | '---------' + | Binary protocol Json protocol + .-----------------. | | + | Compilation | v v + | database (json) | .------------.------------. + '-----------------' | Thrift API | Thrift API | + ^ '------------'------------' + | | Report | Report | + | | storage | viewer | + .---------------. | server | server | + | Buildlogger / | '------------'------------' + | CMake | ^ ^ + '---------------' | | + SQLAlchemy ORM + | | + v v + + SQLite/PostgreSQL + _.-----._ + .- -. + |-_ _-| + | ~-----~ | + | | + `._ _.' + "-----" +``` +## Buildlogger / CMake + +## Buildaction job list +Based on the Compilation database a Buildaction is created for each analyzer. + +## Analyzer client +Multiple analyzer clients can run parallel. +Each Analyzer client: + - processes one Buildaction + - constructs the analysis command + - runs an analyzer + - postprocesses analysis results if required + - sends the analisys results trough Thift binary protocol to the Report storage server + +In quickcheck mode the results are only printed to the standard output. Nothing is stored into a database. + +## Report Storage server +- Provides a Thrift API for result storage. +- Stores the analysis results to the database. +- Detects duplicate results (result in header file detected by multiple analyzer runs). +- Handles that source file contents are stored only once. +- Uses SQLAlchemy to connect to a database backend. + +## Database +- Store multiple analyzer run results. +- Data can be used to generate analysis statistics. + +## Report viewer server +- Multiple clients can connect simultaneously to the report viewer server. +- Uses SQLAlchemy to connect to a database backend. +- Provides a simple https webserver to view documentation. + +## Command line client +- Simple client to view/compare results. +- Can be used for automated scripts for result processing. +- Provides plaintext and json output. + +## Web browser client +- Client to view/filter/compare analysis results. +- Results are dynamically rendered based on the database content. Index: tools/codechecker/docs/checker_docs.md =================================================================== --- /dev/null +++ tools/codechecker/docs/checker_docs.md @@ -0,0 +1,6 @@ + +Checker documentation +---------------------- +Checker doc map is required by the viewer server to show documentation for each checker. +This map contains a checker name and the corresponding md documentation file name. +\include checker_docs/checker_doc_map.json Index: tools/codechecker/docs/checker_docs/checker_doc_map.json =================================================================== --- /dev/null +++ tools/codechecker/docs/checker_docs/checker_doc_map.json @@ -0,0 +1,3 @@ +{ + "core.DivideZero": "core_division_by_zero.md" +} Index: tools/codechecker/docs/checker_docs/core_division_by_zero.md =================================================================== --- /dev/null +++ tools/codechecker/docs/checker_docs/core_division_by_zero.md @@ -0,0 +1,38 @@ +DivisionByZero +================ + +Description +------------- + +This checker finds bugs when division by zero happens. + +Example +------------- + +~~~~~~{.cpp} +int doubleOrNull(int a) +{ + return a > 0 ? 2*a : 0; +} + +void f() +{ + int a = 0.0; + int b = doubleOrNull(a); + a = a / b; // Division by zero +} +~~~~~~ + +Configuration +------------- +None + +Limitations +------------- +None + +Solution +------------- +Check the divisor for zero. +For example, here is the fix for the previous code. +if (b != 0) a = a / b; Index: tools/codechecker/docs/db_schema_guide.md =================================================================== --- /dev/null +++ tools/codechecker/docs/db_schema_guide.md @@ -0,0 +1,73 @@ +# How to modify the database schema + +CodeChecker is developed in rolling release model so it is important to update +the database schema in a backward compatible way. This is achieved using the +[Alembic](http://alembic.readthedocs.org/en/latest/index.html) database +migration tool. + +This is a step-by-step guide, how to modify the schema in a backward compatible +way using migration scripts. + +## Step 1: Update the database model + +CodeChecker uses [SQLAlchemy](http://www.sqlalchemy.org/) for database +operations. You can find the database model in libcodechecker/db_model/orm_model.py file. When +you want to modify the database schema you should update this file first. + +Please read [SQLAlchemy declarative syntax documentation](http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/) +for syntax and semantics. + +## Step 2: Write a migration script + +When you change the database schema, it's essential to write a migration script. +When you start CodeChecker with an existing database, it will automatically +migrate the schema to the latest version using Alembic migration scripts. + +You can write the migration script manually or you can use Alembic's +'autogenerate' feature. + +### Step 2.A: Generating migration scripts using autogenerate + +Alembic can compare the table metadata against the status of a database and +based on this comparison it generates the migration script. Even though this +method has it's [limitations]( +http://alembic.readthedocs.org/en/latest/autogenerate.html#what-does-autogenerate-detect-and-what-does-it-not-detect) +, in most cases it works well. + +To generate a migration script, do the following steps: +1. Start a database with the original, unmodified schema. +2. Go to CodeChecker's source root +3. Edit the sqlalchemy.url option in [alembic.ini]( + http://alembic.readthedocs.org/en/latest/tutorial.html#editing-the-ini-file) + according to your database configuration. +4. Generate the migration script using alembic: + ``` + alembic revision --autogenerate -m "Change description" + ``` +5. The new migration script db_migrate/versions/{hash}_change_description.py is + generated. **You must always check the generated script because sometimes it + isn't correct.** +6. Run all test cases. **All unit tests must pass**. +7. Don't forget to commit the migration script with your other changes. + +Further reading: +- [Auto Generating Migrations](http://alembic.readthedocs.org/en/latest/autogenerate.html) +- [Alembic tutorial](http://alembic.readthedocs.org/en/latest/tutorial.html) +- [Alembic Operation Reference](http://alembic.readthedocs.org/en/latest/ops.html) + +### Step 2.B: Writing migration scripts by hand + +Navigate to the root directory of CodeChecker source code and create an empty +migration script using `alembic revision`: + +``` +alembic revision -m "Change description" +``` + +The new file db_migrate/versions/{hash}_change_description.py is generated. This +file contains an empty `upgrade` and a `downgrade` function. You should always +implement the `upgrade` function. Downgrading is not supported right now. + +You should also read the [Alembic tutorial](http://alembic.readthedocs.org/en/latest/tutorial.html#create-a-migration-script) +and the [Operation Reference](http://alembic.readthedocs.org/en/latest/ops.html) +for details. Index: tools/codechecker/docs/deps.md =================================================================== --- /dev/null +++ tools/codechecker/docs/deps.md @@ -0,0 +1,33 @@ + +### Packaging requirements + * [Git](https://git-scm.com/) (> 1.9.1) + * [Thrift compiler](https://thrift.apache.org/) (> 0.9.2) required to generate python and javascript files + * [Doxygen](http://www.stack.nl/~dimitri/doxygen/) (> 1.8) to generate documentation + * Build logging + - It is possible to build package without the ld-logger. In that case no automatic compilation database generation is done. To build ld-logger 32 and 64 bit versions automatically, `gcc multilib` and `make` is required. + - Compilation command database can be generated with CMake during the build (run `cmake` with the 'CMAKE_EXPORT_COMPILE_COMMANDS' option). CodeChecker can process the generated compilation database at runtime. + + * Other external dependencies are automatically downloaded and + copied to the necessary directories in the package. + +### Runtime requirements +#### Basic +Javascript dependencies are automatically downloaded based on the ext_source_deps_config.json file during package build. + * [Clang Static analyzer](http://clang-analyzer.llvm.org/) (latest stable or [trunk](http://clang.llvm.org/get_started.html)) + * [Clang Tidy](http://clang.llvm.org/extra/clang-tidy/) (available in the clang tools extra repository [trunk](http://clang.llvm.org/get_started.html)) + * [Python2](https://www.python.org/) (> 2.7) + * [Alembic](https://pypi.python.org/pypi/alembic) (>= 0.8.2) database migration support is available only for PostgreSQL database + * [SQLAlchemy](http://www.sqlalchemy.org/) (>= 1.0.9) Python SQL toolkit and Object Relational Mapper, for supporting multiple database backends + - [PyPi SQLAlchemy](https://pypi.python.org/pypi/SQLAlchemy) (> 1.0.2) + * Thrift python modules. Cross-language service building framework to handle data transfer for report storage and result viewer clients + - [PyPi thrift](https://pypi.python.org/pypi/thrift/0.9.2)(> 0.9.2 ) + * [Codemirror](https://codemirror.net/) (MIT) - view source code in the browser + * [Jsplumb](https://jsplumbtoolkit.com/) (community edition, MIT) - draw bug paths + * [Marked](https://github.com/chjj/marked) (BSD) - view documentation for checkers written in markdown (generated dynamically) + * [Dojotoolkit](https://dojotoolkit.org/) (BSD) - main framework for the web UI + * [Highlightjs](https://highlightjs.org/) (BSD) - required for highlighting the source code + +#### PostgreSQL +For the additional PostgreSQL dependecies see the [PostgreSQL setup](postgresql_setup.md) documentation. + +\include ext_source_deps_config.json Index: tools/codechecker/docs/package_layout.md =================================================================== --- /dev/null +++ tools/codechecker/docs/package_layout.md @@ -0,0 +1,15 @@ +# Codechecker package builder + +Short description for the package layout of the generic codechecker package. +Package creation is based on the package layout config file. +It has two main parts a static and a runtime part. + +## Static section +Static part is used to create the main package skeleton (directory structure) where the CodeChecker finds the required files. + +### External checker libraries +External checker libraries can be used in the package. The shared object files should be in the plugin directory and will be automatically loaded at runtime. + +## Runtime section +The runtime part contains informations which will be used only at runtime +to find files during the checking process. Index: tools/codechecker/docs/postgresql_setup.md =================================================================== --- /dev/null +++ tools/codechecker/docs/postgresql_setup.md @@ -0,0 +1,71 @@ + +## PostgreSQL + +## Extra runtime requirements for PostgreSQL database support + * [PostgreSQL](http://www.postgresql.org/ "PostgreSQL") (> 9.3.5) (optional) + * [psycopg2](http://initd.org/psycopg/ "psycopg2") (> 2.5.4) or [pg8000](https://github.com/mfenniak/pg8000 "pg8000") (>= 1.10.0) at least one database connector is required for postgreSQL database support (both are supported) + - [PyPi psycopg2](https://pypi.python.org/pypi/psycopg2/2.6.1) __requires lbpq!__ + - [PyPi pg8000](https://pypi.python.org/pypi/pg8000) + +## Install & setup additional dependencies +Tested on Ubuntu LTS 14.04.2 +~~~~~~{.sh} + +# get the extra postgresql packages +sudo apt-get install libpq-dev python-dev postgresql postgresql-client-common postgresql-common + +# Note: The following PostgreSQL specific steps are only needed when PostgreSQL +# is used for checking. By default CodeChecker uses SQLite. + +# setup database for a test_user +sudo -i -u postgres +# add a test user with "test_pwd" password +createuser --createdb --login --pwprompt test_user +exit + +# PostgreSQL authentication +# PGPASSFILE environment variable should be set to a pgpass file +# For format and further information see PostgreSQL documentation: +# http://www.postgresql.org/docs/current/static/libpq-pgpass.html + +echo "*:5432:*:test_user:test_pwd" >> ~/.pgpass +chmod 0600 ~/.pgpass + +# activate the already created virtualenv in the basic setup +source ~/checker_env/bin/activate + +# install required python modules +pip install -r .ci/python_requirements + +# create codechecker package +./build_package.py -o ~/codechecker_package +cd .. +~~~~~~ + +## Run CodeChecker + +Activate virtualenv. +~~~~~~{.sh} +source ~/checker_env/bin/activate +~~~~~~ + +Add package bin directory to PATH. +This step can be skipped if you always give the path of CodeChecker command. +~~~~~~{.sh} +export PATH=~/codechecker_package/CodeChecker/bin:$PATH +~~~~~~ + +Check a test project. +~~~~~~{.sh} +CodeChecker check --dbusername test_user --postgresql -n test_project_check -b "cd my_test_project && make clean && make" +~~~~~~ + +Start web server to view the results. +~~~~~~{.sh} +CodeChecker server --dbusername test_user --postgresql +~~~~~~ + +View the results with firefox. +~~~~~~{.sh} +firefox http://localhost:8001 +~~~~~~ Index: tools/codechecker/docs/usage.md =================================================================== --- /dev/null +++ tools/codechecker/docs/usage.md @@ -0,0 +1,88 @@ +#CodeChecker command line examples +###1. Quickly check some files and print results in command line +``` +CodeChecker quickcheck -b make +``` +###2. Check a project, update earlier results and view results from a browser +Runs make, logs build, run analyzers and store the results in sqlite db. +``` +CodeChecker check -b make +``` + +Start webserver, which listens on default port localhost:8001 +results can be viewed in a browser +``` +CodeChecker server +``` + +The developer may also suppress false positives +At the end, the project can be rechecked. +``` +CodeChecker check make +``` +###3. Same as use case 2., but the developer would like to enable alpha checkers and llvm checkers +``` +CodeChecker check -e alpha -e llvm -b make +``` +###4. Same as use case 2., but the developer stores suppressed false positives in a text file that is checked in into version control +``` +CodeChecker check –u /home/myuser/myproject/suppress.list -b make +CodeChecker server –u /home/myuser/myproject/suppress.list + +#suppress list will be stored in suppress.list with the bug hashes in the file +#the user checks in the suppress.list file with the source code +``` +###5. Same as use case 2., but the developer wants to give extra Config to clang-tidy or clang-sa +``` +CodeChecker check --saargs clangsa.config --tidyargs clang-tidy.config -b make + +#clang-sa and clang-tidy parameters are stored in simple text files, +#is expected to be in the same format as clang-sa/tidy +#command line parameters and will be passed to every clang-sa/tidy calls +``` + +###6. Asking for command line help for the check subcommand (all other subcommands would be the same: server, checkers,cmd…) +``` +CodeChecker check -h +``` + + +###7. Run analysis on 2 versions of the project +Analyze a large project from a script/Jenkins job periodically. Developers view the results on a central web-server. +If a hit is false positive, developers can mark it and comment it on the web interface and the suppress hashes are stored in a text file that can be version controlled. +Different versions of the project can be compared for new/resolved/unresolved bugs. Differences between runs can be viewed in the web browser or from command line (and can be sent in email if needed). + +Large projects should use postgresql for performance reasons. + +``` +CodeChecker check -n myproject_v1 –postgresql -b make +CodeChecker check -n myproject_v2 –postgresql -b make + +#runs analysis, assumes that postgres is available on the default 5432 TCP port, +#codechecker db user exists and codechecker db can be created +#please note that this use case is also supported with SQLITE db +``` + +Start the webserver and view the diff between the two results in the web browser +``` +CodeChecker server –postgresql + +firefox localhost:8001 +``` +Start the webserver and view the diff between the two results in command line +``` +CodeChecker cmd diff –b myproject_v1 –n myproject_v2 –p 8001 -new + +#assumes that the server is started on port 8001 +#then shows the new bugs in myproject_v2 compared to baseline myproject_v1 + +CodeChecker cmd diff –b myproject_v1 –n myproject_v2 –p 8001 -unresolved + +#assumes that the server is started on port 8001 +#then shows the unresolved bugs in myproject_v2 compared to baseline myproject_v1 + +CodeChecker cmd diff –b myproject_v1 –n myproject_v2 –p 8001 -resolved + +#assumes that the server is started on port 8001 +#then shows the resolved bugs in myproject_v2 compared to baseline myproject_v1 +``` Index: tools/codechecker/docs/user_guide.md =================================================================== --- /dev/null +++ tools/codechecker/docs/user_guide.md @@ -0,0 +1,542 @@ +# CodeChecker Userguide + +## CodeChecker usage + +First of all, you have to setup the environment for CodeChecker. +Codechecker server uses SQLite database (by default) to store the results which is also packed into the package. + +The next step is to start the CodeChecker main script. +The main script can be started with different options. + +~~~~~~~~~~~~~~~~~~~~~ +usage: CodeChecker.py [-h] {check,log,checkers,server,cmd,debug} ... + +Run the codechecker script. + +positional arguments: + {check,log,checkers,server,cmd,debug} + commands + check Run codechecker for a project. + log Build the project and only create a log file (no + checking). + checkers List available checkers. + server Start the codechecker database server. + cmd Command line client + debug Create debug logs for failed actions + +optional arguments: + -h, --help show this help message and exit +~~~~~~~~~~~~~~~~~~~~~ + + +## Default configuration: + +Used ports: +* 5432 - PostgreSQL +* 8001 - CodeChecker result viewer + +## 1. log mode: + +Just build your project and create a log file but do not invoke the source code analysis. + +~~~~~~~~~~~~~~~~~~~~~ +$CodeChecker log --help +usage: CodeChecker.py log [-h] -o LOGFILE -b COMMAND + +optional arguments: + -h, --help show this help message and exit + -o LOGFILE, --output LOGFILE + Path to the log file. + -b COMMAND, --build COMMAND + Build command. +~~~~~~~~~~~~~~~~~~~~~ + +You can change the compilers that should be logged. +Set CC_LOGGER_GCC_LIKE environment variable to a colon separated list. +For example (default): + +~~~~~~~~~~~~~~~~~~~~~ +export CC_LOGGER_GCC_LIKE="gcc:g++:clang" +~~~~~~~~~~~~~~~~~~~~~ + +Example: + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker log -o ../codechecker_myProject_build.log -b "make -j2" +~~~~~~~~~~~~~~~~~~~~~ + +Note: +In case you want to analyze your whole project, do not forget to clean your build tree before logging. + +## 2. check mode: + +### Basic Usage + +Database and connections will be automatically configured. +The main script starts and setups everything what is required for analyzing a project (database server, tables ...). + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker check -w codechecker_workspace -n myTestProject -b "make" +~~~~~~~~~~~~~~~~~~~~~ + +Static analysis can be started also by using an already generated buildlog (see log mode). +If log is not available the analyzer will automatically create it. +An already created CMake json compilation database can be used as well. + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker check -w ~/codechecker_wp -n myProject -l ~/codechecker_wp/build_log.json +~~~~~~~~~~~~~~~~~~~~~ + +### Advanced Usage + +~~~~~~~~~~~~~~~~~~~~~ +usage: CodeChecker check [-h] [-w WORKSPACE] -n NAME (-b COMMAND | -l LOGFILE) + [-j JOBS] [-u SUPPRESS] [-c [DEPRECATED]] + [--update [DEPRECATED]] [--force] [-s SKIPFILE] + [-e ENABLE] [-d DISABLE] [--keep-tmp] + [--analyzers ANALYZERS [ANALYZERS ...]] + [--saargs CLANGSA_ARGS_CFG_FILE] + [--tidyargs TIDY_ARGS_CFG_FILE] [--sqlite] + [--dbport DBPORT] [--dbaddress DBADDRESS] + [--dbname DBNAME] [--dbusername DBUSERNAME] +optional arguments: + -h, --help show this help message and exit + -w WORKSPACE, --workspace WORKSPACE + Directory where the codechecker can store analysis + related data. (default: /home//.codechecker) + -n NAME, --name NAME Name of the analysis. + -b COMMAND, --build COMMAND + Build command which is used to build the project + -l LOGFILE, --log LOGFILE + Path to the log file which is created during the + build. If there is an already generated log file with + the compilation commands generated by 'CodeChecker + log' or 'cmake -DCMAKE_EXPORT_COMPILE_COMMANDS' + CodeChecker check can use it for the analisys in that + case running the original build will be left out from + the analysis process (no log is needed). + -j JOBS, --jobs JOBS Number of jobs. Start multiple processes for faster + analisys (default: 1) + -u SUPPRESS, --suppress SUPPRESS + Path to suppress file. Suppress file can be used to + suppress analysis results during the analisys. It is + based on the bug identifier generated by the compiler + which is experimental. Do not depend too much on this + file because identifier or file format can be changed. + For other in source suppress features see the user + guide. + -c [DEPRECATED], --clean [DEPRECATED] + DEPRECATED argument! (default: None) + --update [DEPRECATED] + DEPRECATED argument! (default: None) + --force Delete analisys results form the database if a run + with the given name already exists (default: False) + -s SKIPFILE, --skip SKIPFILE + Path to skip file. + -e ENABLE, --enable ENABLE + Enable checker. + -d DISABLE, --disable DISABLE + Disable checker. + --keep-tmp Keep temporary report files generated during the + analysis. (default: False) + --analyzers ANALYZERS [ANALYZERS ...] + Select which analyzer should be enabled. Currently + supported analyzers are: clangsa clang-tidy e.g. '-- + analyzers clangsa clang-tidy' (default: ['clangsa', + 'clang-tidy']) + --saargs CLANGSA_ARGS_CFG_FILE + File with arguments which will be forwarded directly + to the Clang static analyzer without modifiaction + --tidyargs TIDY_ARGS_CFG_FILE + File with arguments which will be forwarded directly + to the Clang tidy analyzer without modifiaction + --sqlite [DEPRECATED] + DEPRECATED argument! (default: None) + --postgresql Use PostgreSQL database. (default: False) + --dbport DBPORT Postgres server port. (default: 5432) + --dbaddress DBADDRESS + Postgres database server address (default: localhost) + --dbname DBNAME Name of the database. (default: codechecker) + --dbusername DBUSERNAME + Database user name. (default: codechecker) +~~~~~~~~~~~~~~~~~~~~~ + +CodeChecker is able to handle several analyzer tools. Currently CodeChecker +supports Clang Static Analyzer and Clang Tidy. ```CodeChecker checkers``` +command lists all checkers from each analyzers. These can be switched on and off +by ```-e``` and ```-d``` flags. Furthermore ```--analyzers``` specifies which +analyzer tool should be used (both by default). The tools are completely +independent, so either can be omitted if not present as these are provided by +different binaries. + +#### Forward compiler options + +These options can modify the compilation actions logged by the build logger or +created by cmake (exporting compile commands). The extra compiler options can be +given in config files which are provided by the flags described below. + +The config files can contain placeholders in `$(ENV_VAR)` format. If the +`ENV_VAR` environment variable is set then the placeholder is replaced to its +value. Otherwise an error message is logged saying that the variable is not set, +and in this case an empty string is inserted in the place of the placeholder. + +##### Clang Static Analyzer + +Use the ```--saargs``` argument to a file which contains compilation options. + +```` +CodeChecker check --saargs extra_compile_flags -n myProject -b "make -j4" +```` + +Where the extra_compile_flags file contains additional compilation options. + +Config file example: +``` +-I~/include/for/analysis -I$(MY_LIB)/include -DDEBUG +``` +where `MY_LIB` is the path of a library code. + +##### Clang-tidy + +Use the ```--tidyargs``` argument to a file which contains compilation options. + +```` +CodeChecker check --tidyargs extra_tidy_compile_flags -n myProject -b "make -j4" +```` + +Where the extra_compile_flags file contains additional compilation flags. +Clang tidy requires a different format to add compilation options. +Compilation options can be added before ( ```-extra-arg-before=``` ) and +after (```-extra-arg=```) the original compilation options. + +Config file example: +``` +-extra-arg-before='-I~/include/for/analysis' -extra-arg-before='-I~/other/include/for/analysis/' -extra-arg-before='-I$(MY_LIB)/include' -extra-arg='-DDEBUG' +``` +where `MY_LIB` is the path of a library code. + +### Using SQLite for database: + +CodeChecker can also use SQLite for storing the results. In this case the +SQLite database will be created in the workspace directory. + +In order to use PostgreSQL instead of SQLite, use the ```--postgresql``` command +line argument for ```CodeChecker server``` and ```CodeChecker check``` +commands. If ```--postgresql``` is not given then SQLite is used by default in +which case ```--dbport```, ```--dbaddress```, ```--dbname```, and +```--dbusername``` command line arguments are ignored. + +#### Note: +Schema migration is not supported with SQLite. This means if you upgrade your +CodeChecker to a newer version, you might need to re-check your project. + +### Suppression in the source: + +Suppress comments can be used in the source to suppress specific or all checker results found in a source line. +Suppress comment should be above the line where the bug was found no empty lines are allowed between the line with the bug and the suppress comment. +Only comment lines staring with "//" are supported + +Supported comment formats: + +~~~~~~~~~~~~~~~~~~~~~ +void test() { + int x; + // codechecker_suppress [deadcode.DeadStores] suppress deadcode + x = 1; // warn +} +~~~~~~~~~~~~~~~~~~~~~ + +~~~~~~~~~~~~~~~~~~~~~ +void test() { + int x; + // codechecker_suppress [all] suppress all checker results + x = 1; // warn +} +~~~~~~~~~~~~~~~~~~~~~ + +~~~~~~~~~~~~~~~~~~~~~ +void test() { + int x; + + // codechecker_suppress [all] suppress all + // checker resuls + // with a long + // comment + x = 1; // warn +} +~~~~~~~~~~~~~~~~~~~~~ + +### Suppress file: + +~~~~~~~~~~~~~~~~~~~~~ +-u SUPPRESS +~~~~~~~~~~~~~~~~~~~~~ + +Suppress file can contain bug hashes and comments. +Suppressed bugs will not be showed in the viewer by default. +Usually a reason to suppress a bug is a false positive result (reporting a non-existent bug). Such false positives should be reported, so we can fix the checkers. +A comment can be added to suppressed reports that describes why that report is false positive. You should not edit suppress file by hand. The server should handle it. +The suppress file can be checked into the source code repository. +Bugs can be suppressed on the viewer even when suppress file was not set by command line arguments. This case the suppress will not be permanent. For this reason it is +advised to always provide (the same) suppress file for the checks. + +### Skip file: + +~~~~~~~~~~~~~~~~~~~~~ +-s SKIPFILE, --skip SKIPFILE +~~~~~~~~~~~~~~~~~~~~~ +With a skip file you can filter which files should or shouldn't be checked. +Each line in a skip file should start with a '-' or '+' character followed by a path glob pattern. A minus character means that if a checked file path - including the headers - matches with the pattern, the file will not be checked. The plus character means the opposite: if a file path matches with the pattern, it will be checked. +CodeChecker reads the file from top to bottom and stops at the first matching pattern. + +For example: + +~~~~~~~~~~~~~~~~~~~~~ +-/skip/all/source/in/directory* +-/do/not/check/this.file ++/dir/check.this.file +-/dir/* +~~~~~~~~~~~~~~~~~~~~~ + +### Enable/Disable checkers + +~~~~~~~~~~~~~~~~~~~~~ +-e ENABLE, --enable ENABLE +-d DISABLE, --disable DISABLE +~~~~~~~~~~~~~~~~~~~~~ +You can enable or disable checkers or checker groups. If you want to enable more checker groups use -e multiple times. To get the actual list of checkers run ```CodeChecer checkers``` command. +For example if you want to enable core and security checkers, but want to disable alpha checkers use + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker check -e core -e security -d alpha ... +~~~~~~~~~~~~~~~~~~~~~ + +### Multithreaded Checking + +~~~~~~~~~~~~~~~~~~~~~ +-j JOBS, --jobs JOBS Number of jobs. +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker will execute analysis on as many threads as specified after -j argument. + + +### Various deployment possibilities + +The codechecker server can be started separately when desired. +In that case multiple clients can use the same database to store new results or view old ones. + + +#### Codechecker server and database on the same machine + +Codechecker server and the database are running on the same machine but the database server is started manually. +In this case the database handler and the database can be started manually by running the server command. +The workspace needs to be provided for both the server and the check commands. + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker server -w ~/codechecker_wp --dbname myProjectdb --dbport 5432 --dbaddress localhost --view-port 8001 +~~~~~~~~~~~~~~~~~~~~~ + +The checking process can be started separately on the same machine + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker check -w ~/codechecker_wp -n myProject -b "make -j 4" --dbname myProjectdb --dbaddress localhost --dbport 5432 +~~~~~~~~~~~~~~~~~~~~~ + +or on a different machine + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker check -w ~/codechecker_wp -n myProject -b "make -j 4" --dbname myProjectdb --dbaddress 192.168.1.1 --dbport 5432 +~~~~~~~~~~~~~~~~~~~~~ + + +#### Codechecker server and database are on different machines + +It is possible that the codechecker server and the PostgreSQL database that contains the analysis results are on different machines. To setup PostgreSQL see later section. + +In this case the codechecker server can be started using the following command: + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker server --dbname myProjectdb --dbport 5432 --dbaddress 192.168.1.2 --view-port 8001 +~~~~~~~~~~~~~~~~~~~~~ + +Start codechecker server locally which connects to a remote database (which is started separately). Workspace is not required in this case. + + +Start the checking as explained previously. + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker check -w ~/codechecker_wp -n myProject -b "make -j 4" --dbname myProjectdb --dbaddress 192.168.1.2 --dbport 5432 +~~~~~~~~~~~~~~~~~~~~~ + +## 3. Quick check mode: + +It's possible to quickly check a small project (set of files) for bugs without +storing the results into a database. In this case only the build command is +required and the defect list appears on the console. The defect list doesn't +shows the bug paths by default but you can turn it on using the --steps command +line parameter. + +Basic usage: + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker quickcheck -b 'make' +~~~~~~~~~~~~~~~~~~~~~ + +Enabling bug path: + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker quickcheck -b 'make' --steps +~~~~~~~~~~~~~~~~~~~~~ + +Usage: + +~~~~~~~~~~~~~~~~~~~~~ +usage: CodeChecker.py quickcheck [-h] (-b COMMAND | -l LOGFILE) [-e ENABLE] + [-d DISABLE] [-s] + +optional arguments: + -h, --help show this help message and exit + -b COMMAND, --build COMMAND + Build command. + -l LOGFILE, --log LOGFILE + Path to the log file which is created during the + build. + -e ENABLE, --enable ENABLE + Enable checker. + -d DISABLE, --disable DISABLE + Disable checker. + -s, --steps Print steps. +~~~~~~~~~~~~~~~~~~~~~ + +## 4. checkers mode: + +List all available checkers. + +The ```+``` (or ```-```) sign before a name of a checker shows whether the checker is enabled (or disabled) by default. + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker checkers +~~~~~~~~~~~~~~~~~~~~~ + + +## 5. cmd mode: + +A lightweigh command line interface to query the results of an analysis. +It is a suitable client to integrate with continous integration, schedule maintenance tasks and verifying correct analysis process. +The commands always need a viewer port of an already running CodeChecker server instance (which can be started using CodeChecker server command). + +~~~~~~~~~~~~~~~~~~~~~ +usage: CodeChecker.py cmd [-h] {runs,results,sum,del} ... + +positional arguments: + {runs,results,diff,sum,del} + runs Get the run data. + results List results. + diff Diff two run. + sum Sum results. + del Remove run results. + +optional arguments: + -h, --help show this help message and exit +~~~~~~~~~~~~~~~~~~~~~ + +## 6. debug mode: + +In debug mode CodeChecker can generate logs for failed build actions. The logs can be helpful debugging the checkers. + +## Example Usage + +### Checking files + +Checking with some extra checkers disabled and enabled + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker check -w ~/Develop/workspace -j 4 -b "cd ~/Libraries/myproject && make clean && make -j4" -s ~/Develop/skip.list -u ~/Develop/suppress.txt -e unix.Malloc -d core.uninitialized.Branch -n MyLittleProject -c --dbport 5432 --dbname cctestdb +~~~~~~~~~~~~~~~~~~~~~ + +### View results + +To view the results CodeChecker sever needs to be started. + +~~~~~~~~~~~~~~~~~~~~~ +CodeChecker server -w ~/codes/package/checker_ws/ --dbport 5432 --dbaddress localhost +~~~~~~~~~~~~~~~~~~~~~ + +After the server has started open the outputed link to the browser (localhost:8001 in this example). + +~~~~~~~~~~~~~~~~~~~~~ +[11318] - WARNING! No suppress file was given, suppressed results will be only stored in the database. +[11318] - Checking for database +[11318] - Database is not running yet +[11318] - Starting database +[11318] - Waiting for client requests on [localhost:8001] +~~~~~~~~~~~~~~~~~~~~~ + +### Run CodeChecker distributed in a cluster + +You may want to configure codechecker to do the analysis on separate machines in a distributed way. +Start the postgres database on a central machine (in this example it is called codechecker.central) on a remotely accessible address and port and then run +```CodeChecker check``` on multiple machines (called host1 and host2), specify the remote dbaddress and dbport and use the same run name. + +Create and start an empty database to which the codechecker server can connect. + +#### Setup PostgreSQL (one time only) + +Before the first use, you have to setup PostgreSQL. +PostgreSQL stores its data files in a data directory, so before you start the PostgreSQL server you have to create and init this data directory. +I will call the data directory to pgsql_data. + +Do the following steps: + +~~~~~~~~~~~~~~~~~~~~~ +# on machine codechecker.central + +mkdir -p /path/to/pgsql_data +initdb -U codechecker -D /path/to/pgsql_data -E "SQL_ASCII" +# Start PostgreSQL server on port 5432 +postgres -U codechecker -D /path/to/pgsql_data -p 5432 &>pgsql_log & +~~~~~~~~~~~~~~~~~~~~~ + +#### Run CodeChecker on multiple hosts + +Then you can run codechecker on multiple hosts but using the same run name (in this example this is called "distributed_run". +postgres is listening on codechecker.central port 9999. + +~~~~~~~~~~~~~~~~~~~~~ +# On host1 we check module1 +CodeChecker check -w /tmp/codechecker_ws -b "cd module_1;make" --dbport 5432 --dbaddress codechecker.central -n distributed_run + +# On host2 we check module2 +CodeChecker check -w /tmp/codechecker_ws -b "cd module_2;make" --dbport 5432 --dbaddress codechecker.central -n disributed_run +~~~~~~~~~~~~~~~~~~~~~ + + +#### PostgreSQL authentication (optional) + +If a CodeChecker is run with a user that needs database authentication, the +PGPASSFILE environment variable should be set to a pgpass file +For format and further information see PostgreSQL documentation: +http://www.postgresql.org/docs/current/static/libpq-pgpass.html + +## Debugging CodeChecker + +Environment variables can be used to turn on CodeChecker debug mode. + +Turn on analysis related logging + +~~~~~~~~~~~~~~~~~~~~~ +export CODECHECKER_VERBOSE=debug_analyzer +~~~~~~~~~~~~~~~~~~~~~ + +Turn on CodeChecker debug level logging + +~~~~~~~~~~~~~~~~~~~~~ +export CODECHECKER_VERBOSE=debug +~~~~~~~~~~~~~~~~~~~~~ + +If debug logging is enabled and PostgreSQL database is used, PostgreSQL logs are written to postgresql.log in the workspace directory. + +Turn on SQL_ALCHEMY debug level logging + +~~~~~~~~~~~~~~~~~~~~~ +export CODECHECKER_ALCHEMY_LOG=True +~~~~~~~~~~~~~~~~~~~~~ Index: tools/codechecker/libcodechecker/__init__.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/__init__.py @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. Index: tools/codechecker/libcodechecker/analysis_manager.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analysis_manager.py @@ -0,0 +1,197 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +''' + +import os +import sys +import signal +import multiprocessing +import ntpath +import traceback +import shutil +from collections import defaultdict + +from libcodechecker import logger +from libcodechecker import analyzer_env + +from libcodechecker.analyzers import analyzer_types + +LOG = logger.get_new_logger('ANALISYS MANAGER') + + +def worker_result_handler(results): + """ + print the analisys summary + """ + + successful_analysis = defaultdict(int) + failed_analisys = defaultdict(int) + skipped_num = 0 + + for res, skipped, analyzer_type in results: + if skipped: + skipped_num += 1 + else: + if res == 0: + successful_analysis[analyzer_type] += 1 + else: + failed_analisys[analyzer_type] += 1 + + LOG.info("----==== Summary ====----") + LOG.info('Total compilation commands: ' + str(len(results))) + if successful_analysis: + LOG.info('Successfully analyzed') + for analyzer_type, res in successful_analysis.iteritems(): + LOG.info(' ' + analyzer_type + ': ' + str(res)) + + if failed_analisys: + LOG.info("Failed to analyze") + for analyzer_type, res in failed_analisys.iteritems(): + LOG.info(' ' + analyzer_type + ': ' + str(res)) + + if skipped_num: + LOG.info('Skipped compilation commands: ' + str(skipped_num)) + LOG.info("----=================----") + +def check(check_data): + """ + Invoke clang with an action which called by processes. + Different analyzer object belongs to for each build action + + skiplist handler is None if no skip file was configured + """ + args, action, context, analyzer_config_map, skp_handler, \ + report_output_dir, use_db = check_data + + skipped = False + try: + # if one analysis fails the check fails + return_codes = 0 + skipped = False + for source in action.sources: + + # if there is no skiplist handler there was no skip list file + # in the command line + # cpp file skipping is handled here + _, source_file_name = ntpath.split(source) + + if skp_handler and skp_handler.should_skip(source): + LOG.debug_analyzer(source_file_name + ' is skipped') + skipped = True + continue + + # construct analyzer env + analyzer_environment = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + run_id = context.run_id + + rh = analyzer_types.construct_result_handler(args, + action, + run_id, + report_output_dir, + context.severity_map, + skp_handler, + use_db) + + #LOG.info('Analysing ' + source_file_name) + + # create a source analyzer + source_analyzer = analyzer_types.construct_analyzer(action, + analyzer_config_map) + + # source is the currently analyzed source file + # there can be more in one buildaction + source_analyzer.source_file = source + + # fills up the result handler with the analyzer information + source_analyzer.analyze(rh, analyzer_environment) + + if rh.analyzer_returncode == 0: + # analysis was successful + # processing results + if rh.analyzer_stdout != '': + LOG.debug_analyzer('\n' + rh.analyzer_stdout) + if rh.analyzer_stderr != '': + LOG.debug_analyzer('\n' + rh.analyzer_stderr) + rh.postprocess_result() + rh.handle_results() + else: + # analisys failed + LOG.error('Analyzing ' + source_file_name + ' failed.') + if rh.analyzer_stdout != '': + LOG.error(rh.analyzer_stdout) + if rh.analyzer_stderr != '': + LOG.error(rh.analyzer_stderr) + return_codes = rh.analyzer_returncode + + if not args.keep_tmp: + rh.clean_results() + + return (return_codes, skipped, action.analyzer_type) + + except Exception as e: + LOG.debug_analyzer(str(e)) + traceback.print_exc(file=sys.stdout) + return (1, skipped, action.analyzer_type) + +def start_workers(args, actions, context, analyzer_config_map, skp_handler): + """ + start the workers in the process pool + for every buildaction there is worker which makes the analysis + """ + + # Handle SIGINT to stop this script running + def signal_handler(*arg, **kwarg): + try: + pool.terminate() + finally: + sys.exit(1) + + signal.signal(signal.SIGINT, signal_handler) + + # create report output dir this will be used by the result handlers for each + # analyzer to store analyzer results or temporary files + # each analyzer instance does its own cleanup + report_output = os.path.join(context.codechecker_workspace, + args.name + '_reports') + + if not os.path.exists(report_output): + os.mkdir(report_output) + + # Start checking parallel + pool = multiprocessing.Pool(args.jobs) + # pool.map(check, actions, 1) + + try: + # Workaround, equialent of map + # The main script does not get signal + # while map or map_async function is running + # It is a python bug, this does not happen if a timeout is specified; + # then receive the interrupt immediately + + analyzed_actions = [(args, + build_action, + context, + analyzer_config_map, + skp_handler, + report_output, + True ) for build_action in actions] + + pool.map_async(check, + analyzed_actions, + 1, + callback=worker_result_handler).get(float('inf')) + + pool.close() + except Exception: + pool.terminate() + raise + finally: + pool.join() + if not args.keep_tmp: + LOG.debug('Removing temporary directory: ' + report_output) + shutil.rmtree(report_output) Index: tools/codechecker/libcodechecker/analyzer.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzer.py @@ -0,0 +1,156 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +""" +Prepare and start different analisys types +""" +import os +import sys +import json +import time +import copy + +from libcodechecker import client +from libcodechecker import logger +from libcodechecker import analysis_manager +from libcodechecker import skiplist_handler + +from libcodechecker.analyzers import analyzer_types + +LOG = logger.get_new_logger('ANALYZER') + +def prepare_actions(actions, enabled_analyzers): + """ + set the analyzer type for each buildaction + muliply actions if multiple source analyzers are set + """ + res = [] + + for ea in enabled_analyzers: + for action in actions: + new_action = copy.deepcopy(action) + new_action.analyzer_type = ea + res.append(new_action) + return res + + +def run_check(args, actions, context): + """ + prepare: + - analyzer config handlers + - skiplist handling + - analyzer severity levels + + stores analysis related data to the database and starts the analysis + """ + + if args.jobs <= 0: + args.jobs = 1 + + LOG.debug_analyzer("Checking supported analyzers.") + enabled_analyzers = analyzer_types.check_supported_analyzers( + args.analyzers, + context) + + # load severity map from config file + LOG.debug_analyzer("Loading checker severity map.") + if os.path.exists(context.checkers_severity_map_file): + with open(context.checkers_severity_map_file, 'r') as sev_conf_file: + severity_config = sev_conf_file.read() + + context.severity_map = json.loads(severity_config) + + actions = prepare_actions(actions, enabled_analyzers) + + analyzer_config_map = {} + + package_version = context.version['major'] + '.' + context.version['minor'] + + suppress_file = '' + try: + suppress_file = os.path.realpath(args.suppress) + except AttributeError: + LOG.debug_analyzer('Suppress file was not set in the command line') + + + # Create one skip list handler shared between the analysis manager workers + skip_handler = None + try: + if args.skipfile: + LOG.debug_analyzer("Creating skiplist handler.") + skip_handler = skiplist_handler.SkipListHandler(args.skipfile) + except AttributeError: + LOG.debug_analyzer('Skip file was not set in the command line') + + + with client.get_connection() as connection: + context.run_id = connection.add_checker_run(' '.join(sys.argv), + args.name, + package_version, + args.force) + + # clean previous suppress information + client.clean_suppress(connection, context.run_id) + + if os.path.exists(suppress_file): + client.send_suppress(context.run_id, connection, suppress_file) + + analyzer_config_map = analyzer_types.build_config_handlers(args, + context, + enabled_analyzers, + connection) + if skip_handler: + connection.add_skip_paths(context.run_id, + skip_handler.get_skiplist()) + + LOG.info("Static analysis is starting ...") + start_time = time.time() + + analysis_manager.start_workers(args, + actions, + context, + analyzer_config_map, + skip_handler) + + end_time = time.time() + + with client.get_connection() as connection: + connection.finish_checker_run(context.run_id) + + LOG.info("Analysis length: " + str(end_time - start_time) + " sec.") + + +def run_quick_check(args, + context, + actions): + ''' + This function implements the "quickcheck" feature. + No result is stored to a database + ''' + + enabled_analyzers = set() + + enabled_analyzers = analyzer_types.check_supported_analyzers(args.analyzers, + context) + + actions = prepare_actions(actions, enabled_analyzers) + + analyzer_config_map = {} + + analyzer_config_map = analyzer_types.build_config_handlers(args, + context, + enabled_analyzers) + + for action in actions: + check_data = (args, + action, + context, + analyzer_config_map, + None, + args.workspace, + False) + + analysis_manager.check(check_data) Index: tools/codechecker/libcodechecker/analyzer_crash_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzer_crash_handler.py @@ -0,0 +1,77 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +'''Module to handle analyzer crash.''' + +import subprocess +import shlex +import tempfile +import signal + +from libcodechecker import logger + + +class AnalyzerCrashHandler(object): + + def __init__(self, context, analyzer_env): + self._context = context + self._analyzer_env = analyzer_env + self._logger = logger.get_new_logger('ANALYZER_CRASH_HANDLER') + + # -------------------------------------------------------------------------- + def get_crash_info(self, build_cmd): + ''' Get the crash info by running the build command + with gdb again if there was some crash. + ''' + def signal_handler(*args, **kwargs): + try: + result.terminate() + finally: + raise KeyboardInterrupt('CTRL+C') + + signal.signal(signal.SIGINT, signal_handler) + + gdb_cmd = [] + gdb_cmd.append('gdb') + gdb_cmd.append('--batch') + # add gdb script path is in the DT unter the config folder + gdb_cmd.append('--command=' + self._context.gdb_config_file) + gdb_cmd.append('--args') + gdb_cmd.extend(build_cmd) + + tmp_stdout = tempfile.TemporaryFile() + tmp_stderr = tempfile.TemporaryFile() + result = "" + + # gdb uses python3, so it is crashed when any python2 module in PYTHONPATH + # bug: https://bugs.launchpad.net/ubuntu/+source/apport/+bug/1398033 + self._analyzer_env['PYTHONPATH'] = '' + + try: + result = subprocess.Popen(shlex.split(' '.join(gdb_cmd)), + env=self._analyzer_env, + stdout=tmp_stdout, stderr=tmp_stderr) + + result.wait() + + tmp_stdout.seek(0) + tmp_stderr.seek(0) + + output_stdout = tmp_stdout.read() + output_stderr = tmp_stderr.read() + result = output_stdout + '\n' + output_stderr + except KeyboardInterrupt: + raise + except: + result = 'Failed to get extra debug information using gdb:\n' + \ + ' '.join(gdb_cmd) + finally: + tmp_stdout.close() + tmp_stderr.close() + + return result + + # -------------------------------------------------------------------------- Index: tools/codechecker/libcodechecker/analyzer_env.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzer_env.py @@ -0,0 +1,68 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +'''''' + +import os + +from libcodechecker import logger + +LOG = logger.get_new_logger('ENV') + + +# ------------------------------------------------------------------------------ +def get_log_env(logfile, context, original_env): + ''' + Environment for logging. With the ld logger. + Keep the original environment unmodified as possible + Only environment variables required for logging are changed + ''' + new_env = original_env + + new_env[context.env_var_cc_logger_bin] = context.path_logger_bin + + new_env['LD_PRELOAD'] = context.logger_lib_name + + try: + original_ld_library_path = new_env['LD_LIBRARY_PATH'] + new_env['LD_LIBRARY_PATH'] = context.path_logger_lib + \ + ':' + original_ld_library_path + except: + new_env['LD_LIBRARY_PATH'] = context.path_logger_lib + + # set ld logger logfile + new_env[context.env_var_cc_logger_file] = logfile + + return new_env + + +# ----------------------------------------------------------------------------- +def get_check_env(path_env_extra, ld_lib_path_extra): + ''' + Extending the checker environment. + Check environment is extended to find tools if they ar not on + the default places + ''' + new_env = os.environ.copy() + + if len(path_env_extra) > 0: + extra_path = ':'.join(path_env_extra) + LOG.debug_analyzer('Extending PATH environment variable with: ' + extra_path) + + try: + new_env['PATH'] = extra_path + ':' + new_env['PATH'] + except: + new_env['PATH'] = extra_path + + if len(ld_lib_path_extra) > 0: + extra_lib = ':'.join(ld_lib_path_extra) + LOG.debug_analyzer('Extending LD_LIBRARY_PATH environment variable with: ' + extra_lib) + try: + original_ld_library_path = new_env['LD_LIBRARY_PATH'] + new_env['LD_LIBRARY_PATH'] = extra_lib + ':' + original_ld_library_path + except: + new_env['LD_LIBRARY_PATH'] = extra_lib + + return new_env Index: tools/codechecker/libcodechecker/analyzers/__init__.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/__init__.py @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. Index: tools/codechecker/libcodechecker/analyzers/analyzer_base.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/analyzer_base.py @@ -0,0 +1,132 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import sys +import shlex +import signal +import subprocess + +from abc import ABCMeta, abstractmethod + +from libcodechecker import logger + +LOG = logger.get_new_logger('ANALYZER_BASE') + + +class SourceAnalyzer(object): + """ + base class for different source analyzers + """ + __metaclass__ = ABCMeta + + + def __init__(self, config_handler, buildaction): + self.__config_handler = config_handler + self.__buildaction = buildaction + self.__source_file = '' + self.__checkers = [] + + @property + def checkers(self): + return self.__checkers + + @property + def buildaction(self): + return self.__buildaction + + @property + def config_handler(self): + return self.__config_handler + + @property + def source_file(self): + """ + the currently analyzed source file + """ + return self.__source_file + + @source_file.setter + def source_file(self, file_path): + """ + the currently analyzed source file + """ + self.__source_file = file_path + + @abstractmethod + def construct_analyzer_cmd(self, result_handler): + """ + construct the analyzer command + """ + pass + + def analyze(self, res_handler, env=None): + """ + run the analyzer + """ + LOG.debug('Running analyzer ...') + + def signal_handler(*args, **kwargs): + # Clang does not kill its child processes, so I have to + try: + g_pid = proc.pid + os.killpg(g_pid, signal.SIGTERM) + finally: + sys.exit(os.EX_OK) + + signal.signal(signal.SIGINT, signal_handler) + + # NOTICE! + # the currently analyzed source file needs to be set beforer the + # analyzer command is constructed + # the analyzer output file is based on the currently analyzed source + res_handler.analyzed_source_file = self.source_file + + # construct the analyzer cmd + analyzer_cmd = self.construct_analyzer_cmd(res_handler) + + LOG.debug('\n' + ' '.join(analyzer_cmd)) + + res_handler.analyzer_cmd = analyzer_cmd + analyzer_cmd = ' '.join(analyzer_cmd) + try: + ret_code, stdout, stderr = SourceAnalyzer.run_proc(analyzer_cmd, + env) + res_handler.analyzer_returncode = ret_code + res_handler.analyzer_stdout = stdout + res_handler.analyzer_stderr = stderr + return res_handler + + except Exception as ex: + LOG.error(ex) + res_handler.analyzer_returncode = 1 + return res_handler + + @abstractmethod + def get_analyzer_checkers(self, config_handler, env): + """ + return the checkers available in the analyzer + """ + pass + + @staticmethod + def run_proc(command, env=None, cwd=None): + """ + Just run the given command and return the returncode + and the stdout and stderr outputs of the process. + """ + + cmd = shlex.split(command, posix=False) + proc = subprocess.Popen(cmd, + bufsize=-1, + env=env, + preexec_fn=os.setsid, + cwd=cwd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + + (stdout, stderr) = proc.communicate() + return proc.returncode, stdout, stderr Index: tools/codechecker/libcodechecker/analyzers/analyzer_clang_tidy.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/analyzer_clang_tidy.py @@ -0,0 +1,131 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +''' +''' + +import re +import subprocess +import shlex + +from libcodechecker import logger + +from libcodechecker.analyzers import analyzer_base + +LOG = logger.get_new_logger('CLANG TIDY') + + +class ClangTidy(analyzer_base.SourceAnalyzer): + """ + constructs the clang tidy analyzer commands + """ + + def __parse_checkers(self, tidy_output): + """ + parse clang tidy checkers list + skip clang static analyzer checkers + store them to checkers + """ + for line in tidy_output.splitlines(): + line = line.strip() + if re.match(r'^Enabled checks:', line) or line == '': + continue + elif line.startswith('clang-analyzer-'): + continue + else: + match = re.match(r'^\S+$', line) + if match: + self.checkers.append((match.group(0), '')) + + def get_analyzer_checkers(self, config_handler, env): + """ + return the list of the supported checkers + """ + if not self.checkers: + analyzer_binary = config_handler.analyzer_binary + + command = [analyzer_binary] + command.append("-list-checks") + command.append("-checks='*'") + command.append("-") + command.append("--") + + try: + command = shlex.split(' '.join(command)) + result = subprocess.check_output(command, + env=env) + except subprocess.CalledProcessError as cperr: + LOG.error(cperr) + return {} + + self.__parse_checkers(result) + + return self.checkers + + def construct_analyzer_cmd(self, res_handler): + """ + """ + try: + config = self.config_handler + + analyzer_bin = config.analyzer_binary + + analyzer_cmd = [] + analyzer_cmd.append(analyzer_bin) + + # disable all checkers by default + checkers_cmdline = '-*' + + # config handler stores which checkers are enabled or disabled + for checker_name, value in config.checks().iteritems(): + enabled, _ = value + if enabled: + checkers_cmdline += ',' + checker_name + else: + checkers_cmdline += ',-' + checker_name + + analyzer_cmd.append("-checks='" + checkers_cmdline.lstrip(',') + "'") + + LOG.debug(config.analyzer_extra_arguments) + analyzer_cmd.append(config.analyzer_extra_arguments) + + analyzer_cmd.append(self.source_file) + + analyzer_cmd.append("--") + + extra_arguments_before = [] + if len(config.compiler_resource_dirs) > 0: + for inc_dir in config.compiler_resource_dirs: + extra_arguments_before.append('-resource-dir') + extra_arguments_before.append(inc_dir) + extra_arguments_before.append('-isystem') + extra_arguments_before.append(inc_dir) + + if config.compiler_sysroot: + extra_arguments_before.append('--sysroot') + extra_arguments_before.append(config.compiler_sysroot) + + for path in config.system_includes: + extra_arguments_before.append('-isystem') + extra_arguments_before.append(path) + + for path in config.includes: + extra_arguments_before.append('-I') + extra_arguments_before.append(path) + + # Set lang + extra_arguments_before.append('-x') + extra_arguments_before.append(self.buildaction.lang) + + analyzer_cmd.append(' '.join(extra_arguments_before)) + + analyzer_cmd.extend(self.buildaction.analyzer_options) + + return analyzer_cmd + + except Exception as ex: + LOG.error(ex) + return [] Index: tools/codechecker/libcodechecker/analyzers/analyzer_clangsa.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/analyzer_clangsa.py @@ -0,0 +1,168 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import re +import shlex +import subprocess + +from libcodechecker import logger + +from libcodechecker.analyzers import analyzer_base + +LOG = logger.get_new_logger('CLANGSA') + +class ClangSA(analyzer_base.SourceAnalyzer): + """ + constructs clang static analyzer commmands + """ + + def __parse_checkers(self, clangsa_output): + """ + parse clang static analyzer checkers + store them to checkers + """ + + # checker name and description in one line + pattern = re.compile(r'^\s\s(?P\S*)\s*(?P.*)') + + checker_name = None + for line in clangsa_output.splitlines(): + if re.match(r'^CHECKERS:', line) or line == '': + continue + elif checker_name and not re.match(r'^\s\s\S', line): + # collect description for the checker name + self.checkers.append((checker_name, line.strip())) + checker_name = None + elif re.match(r'^\s\s\S+$', line.rstrip()): + # only checker name is in the line + checker_name = line.strip() + else: + # checker name and description is in one line + match = pattern.match(line.rstrip()) + if match: + current = match.groupdict() + self.checkers.append((current['checker_name'], + current['description'])) + + def get_analyzer_checkers(self, config_handler, env): + """ + return the list of the supported checkers + """ + if not self.checkers: + analyzer_binary = config_handler.analyzer_binary + + command = [analyzer_binary, "-cc1"] + for plugin in config_handler.analyzer_plugins: + command.append("-load") + command.append(plugin) + command.append("-analyzer-checker-help") + + try: + command = shlex.split(' '.join(command)) + result = subprocess.check_output(command, + env=env) + except subprocess.CalledProcessError as cperr: + LOG.error(cperr) + return {} + + self.__parse_checkers(result) + + return self.checkers + + def construct_analyzer_cmd(self, res_handler): + """ + called by the analyzer method + construct the analyzer command + """ + try: + # get an putput file from the result handler + analyzer_output_file = res_handler.get_analyzer_result_file() + + analyzer_mode = 'plist-multi-file' + + # get the checkers list from the config_handler + # checker order matters + config = self.config_handler + + analyzer_bin = config.analyzer_binary + + analyzer_cmd = [] + + analyzer_cmd.append(analyzer_bin) + + if len(config.compiler_resource_dirs) > 0: + for inc_dir in config.compiler_resource_dirs: + analyzer_cmd.append('-resource-dir') + analyzer_cmd.append(inc_dir) + analyzer_cmd.append('-isystem') + analyzer_cmd.append(inc_dir) + + # compiling is enough + analyzer_cmd.append('-c') + + analyzer_cmd.append('--analyze') + + # turn off clang hardcoded checkers list + analyzer_cmd.append('--analyzer-no-default-checks') + + + for plugin in config.analyzer_plugins: + analyzer_cmd.append("-Xclang") + analyzer_cmd.append("-plugin") + analyzer_cmd.append("-Xclang") + analyzer_cmd.append("checkercfg") + analyzer_cmd.append("-Xclang") + analyzer_cmd.append("-load") + analyzer_cmd.append("-Xclang") + analyzer_cmd.append(plugin) + + analyzer_cmd.append('-Xclang') + analyzer_cmd.append('-analyzer-opt-analyze-headers') + analyzer_cmd.append('-Xclang') + analyzer_cmd.append('-analyzer-output=' + analyzer_mode) + + if config.compiler_sysroot: + analyzer_cmd.append('--sysroot') + analyzer_cmd.append(config.compiler_sysroot) + + for path in config.system_includes: + analyzer_cmd.append('-isystem') + analyzer_cmd.append(path) + + for path in config.includes: + analyzer_cmd.append('-I') + analyzer_cmd.append(path) + + analyzer_cmd.append('-o') + analyzer_cmd.append(analyzer_output_file) + + # config handler stores which checkers are enabled or disabled + for checker_name, value in config.checks().iteritems(): + enabled, description = value + if enabled: + analyzer_cmd.append('-Xclang') + analyzer_cmd.append('-analyzer-checker=' + checker_name) + else: + analyzer_cmd.append('-Xclang') + analyzer_cmd.append('-analyzer-disable-checker') + analyzer_cmd.append('-Xclang') + analyzer_cmd.append(checker_name) + + # Set lang + analyzer_cmd.append('-x') + analyzer_cmd.append(self.buildaction.lang) + + analyzer_cmd.append(config.analyzer_extra_arguments) + + analyzer_cmd.extend(self.buildaction.analyzer_options) + + analyzer_cmd.append(self.source_file) + + return analyzer_cmd + + except Exception as ex: + LOG.error(ex) + return [] Index: tools/codechecker/libcodechecker/analyzers/analyzer_types.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/analyzer_types.py @@ -0,0 +1,376 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +""" +supported analyzer types +""" +import os +import re +import sys + +from libcodechecker import logger +from libcodechecker import analyzer_env +from libcodechecker import host_check +from libcodechecker import client + +from libcodechecker.analyzers import analyzer_clangsa +from libcodechecker.analyzers import config_handler_clangsa +from libcodechecker.analyzers import result_handler_clangsa + +from libcodechecker.analyzers import analyzer_clang_tidy +from libcodechecker.analyzers import result_handler_clang_tidy +from libcodechecker.analyzers import config_handler_clang_tidy + +LOG = logger.get_new_logger('ANALYZER TYPES') + + +CLANG_SA = 'clangsa' +CLANG_TIDY = 'clang-tidy' + +supported_analyzers = {CLANG_SA, CLANG_TIDY} + +def is_sa_checker_name(checker_name): + """ + match for Clang Static analyzer names like + + unix + unix.Malloc + security.insecureAPI + security.insecureAPI.gets + """ + # no '-' is alowed in the checker name + sa_checker_name = r'^[^-]+$' + ptn = re.compile(sa_checker_name) + + if ptn.match(checker_name): + return True + return False + +def is_tidy_checker_name(checker_name): + """ + match for Clang Tidy analyzer names like + + -* + modernize-* + clang-diagnostic-* + cert-fio38-c + google-global-names-in-headers + """ + # must contain at least one '-' + tidy_checker_name = r'^(?=.*[\-]).+$' + + ptn = re.compile(tidy_checker_name) + + if ptn.match(checker_name): + return True + return False + + +def check_supported_analyzers(analyzers, context): + """ + check if the selected analyzers are supported + """ + + check_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + analyzer_binaries = context.analyzer_binaries + + enabled_analyzers = set() + + for analyzer_name in analyzers: + if analyzer_name not in supported_analyzers: + LOG.error('Unsupported analyzer ' + analyzer_name + ' !') + sys.exit(1) + else: + # get the compiler binary to check if it can run + available_analyzer = True + analyzer_bin = analyzer_binaries.get(analyzer_name) + if not analyzer_bin: + LOG.debug_analyzer('Failed to detect analyzer binary ' + analyzer_name) + available_analyzer = False + if not host_check.check_clang(analyzer_bin, check_env): + LOG.warning('Failed to run analyzer ' + + analyzer_name + ' !') + available_analyzer = False + if available_analyzer: + enabled_analyzers.add(analyzer_name) + + return enabled_analyzers + + +def construct_analyzer_type(analyzer_type, config_handler, buildaction): + """ + construct a specific analyzer based on the type + """ + + if analyzer_type == CLANG_SA: + LOG.debug_analyzer('Constructing clangSA analyzer') + + analyzer = analyzer_clangsa.ClangSA(config_handler, + buildaction) + + return analyzer + + elif analyzer_type == CLANG_TIDY: + LOG.debug_analyzer("Constructing clang-tidy analyzer") + + analyzer = analyzer_clang_tidy.ClangTidy(config_handler, + buildaction) + + return analyzer + else: + LOG.error('Not supported analyzer type') + return None + + +def construct_analyzer(buildaction, + analyzer_config_map): + """ + construct an analyzer + """ + try: + LOG.debug_analyzer('Constructing analyzer') + analyzer_type = buildaction.analyzer_type + # get the proper config handler for this analyzer type + config_handler = analyzer_config_map.get(analyzer_type) + + analyzer = construct_analyzer_type(analyzer_type, + config_handler, + buildaction) + return analyzer + + except Exception as ex: + LOG.debug_analyzer(ex) + return None + +def initialize_checkers(config_handler, + checkers, + default_checkers=None, + cmdline_checkers=None): + + # by default disable all checkers + for checker_name, description in checkers: + config_handler.add_checker(checker_name, False, description) + + # set default enabled or disabled checkers + if default_checkers: + for checker in default_checkers: + for checker_name, enabled in checker.items(): + if enabled: + config_handler.enable_checker(checker_name) + else: + config_handler.disable_checker(checker_name) + + # set user defined enabled or disabled checkers from the + # command line + if cmdline_checkers: + for checker_name, enabled in cmdline_checkers: + if enabled: + config_handler.enable_checker(checker_name) + else: + config_handler.disable_checker(checker_name) + +def __replace_env_var(cfg_file): + def replacer(matchobj): + env_var = matchobj.group(1) + if matchobj.group(1) not in os.environ: + LOG.error(env_var + ' environment variable not set in ' + cfg_file) + return '' + return os.environ[env_var] + return replacer + +def __build_clangsa_config_handler(args, context): + """ + Build the config handler for clang static analyzer + Handle config options from the command line and config files + """ + + config_handler = config_handler_clangsa.ClangSAConfigHandler() + config_handler.analyzer_plugins_dir = context.checker_plugin + config_handler.analyzer_binary = context.analyzer_binaries.get(CLANG_SA) + config_handler.compiler_resource_dirs = context.compiler_resource_dirs + config_handler.compiler_sysroot = context.compiler_sysroot + config_handler.system_includes = context.extra_system_includes + config_handler.includes = context.extra_includes + try: + with open(args.clangsa_args_cfg_file, 'rb') as sa_cfg: + config_handler.analyzer_extra_arguments = \ + re.sub('\$\((.*?)\)', + __replace_env_var(args.clangsa_args_cfg_file), + sa_cfg.read().strip()) + except IOError as ioerr: + LOG.debug_analyzer(ioerr) + except AttributeError as aerr: + # no clangsa arguments file was given in the command line + LOG.debug_analyzer(aerr) + + analyzer = construct_analyzer_type(CLANG_SA, config_handler, None) + + check_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + checkers = analyzer.get_analyzer_checkers(config_handler, check_env) + + # read clang-tidy checkers from the config file + clang_sa_checkers = context.default_checkers_config.get(CLANG_SA + + '_checkers') + try: + cmdline_checkers = args.ordered_checkers + except AttributeError: + LOG.debug_analyzer('No checkers were defined in the command line for' + + CLANG_SA) + cmdline_checkers = None + + initialize_checkers(config_handler, + checkers, + clang_sa_checkers, + cmdline_checkers) + + return config_handler + +def __build_clang_tidy_config_handler(args, context): + """ + Build the config handler for clang tidy analyzer + Handle config options from the command line and config files + """ + + config_handler = config_handler_clang_tidy.ClangTidyConfigHandler() + config_handler.analyzer_binary = context.analyzer_binaries.get(CLANG_TIDY) + config_handler.compiler_resource_dirs = context.compiler_resource_dirs + config_handler.compiler_sysroot = context.compiler_sysroot + config_handler.system_includes = context.extra_system_includes + config_handler.includes = context.extra_includes + + try: + with open(args.tidy_args_cfg_file, 'rb') as tidy_cfg: + config_handler.analyzer_extra_arguments = \ + re.sub('\$\((.*?)\)', __replace_env_var, tidy_cfg.read().strip()) + except IOError as ioerr: + LOG.debug_analyzer(ioerr) + except AttributeError as aerr: + # no clang tidy arguments file was given in the command line + LOG.debug_analyzer(aerr) + + analyzer = construct_analyzer_type(CLANG_TIDY, config_handler, None) + check_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + checkers = analyzer.get_analyzer_checkers(config_handler, check_env) + + # read clang-tidy checkers from the config file + clang_tidy_checkers = context.default_checkers_config.get(CLANG_TIDY + + '_checkers') + try: + cmdline_checkers = args.ordered_checkers + except AttributeError: + LOG.debug_analyzer('No checkers were defined in the command line for ' + + CLANG_TIDY) + cmdline_checkers = None + + initialize_checkers(config_handler, + checkers, + clang_tidy_checkers, + cmdline_checkers) + + + return config_handler + + +def build_config_handlers(args, context, enabled_analyzers, connection=None): + """ + construct multiple config handlers and if there is a connection + store configs into the database + + handle config from command line or from config file if no command line + config is given + + supported command line config format is in JSON tidy supports YAML also but + no standard lib for yaml parsing is available in python + + """ + + run_id = context.run_id + analyzer_config_map = {} + + for ea in enabled_analyzers: + if ea == CLANG_SA: + config_handler = __build_clangsa_config_handler(args, context) + analyzer_config_map[ea] = config_handler + + elif ea == CLANG_TIDY: + config_handler = __build_clang_tidy_config_handler(args, context) + analyzer_config_map[ea] = config_handler + else: + LOG.debug_analyzer('Not supported analyzer type. No configuration handler will be created') + + if connection: + # collect all configuration options and store them together + configs = [] + for _, config_handler in analyzer_config_map.iteritems(): + configs.extend(config_handler.get_checker_configs()) + + client.replace_config_in_db(run_id, connection, configs) + + return analyzer_config_map + + +def construct_result_handler(args, + buildaction, + run_id, + report_output, + severity_map, + skiplist_handler, + store_to_db=False): + """ + construct a result handler + """ + + if store_to_db: + # create a result handler which stores the results into a database + if buildaction.analyzer_type == CLANG_SA: + csa_res_handler = result_handler_clangsa.ClangSAPlistToDB( + buildaction, + report_output, + run_id) + + csa_res_handler.severity_map = severity_map + csa_res_handler.skiplist_handler = skiplist_handler + return csa_res_handler + + elif buildaction.analyzer_type == CLANG_TIDY: + ct_res_handler = result_handler_clang_tidy.ClangTidyPlistToDB( + buildaction, + report_output, + run_id) + + ct_res_handler.severity_map = severity_map + ct_res_handler.skiplist_handler = skiplist_handler + return ct_res_handler + + else: + LOG.error('Not supported analyzer type') + return None + else: + if buildaction.analyzer_type == CLANG_SA: + csa_res_handler = result_handler_clangsa.ClangSAPlistToStdout( + buildaction, + report_output) + + csa_res_handler.print_steps = args.print_steps + return csa_res_handler + + elif buildaction.analyzer_type == CLANG_TIDY: + ct_res_handler = result_handler_clang_tidy.ClangTidyPlistToStdout( + buildaction, + report_output) + + ct_res_handler.severity_map = severity_map + ct_res_handler.skiplist_handler = skiplist_handler + return ct_res_handler + else: + LOG.error('Not supported analyzer type') + return None Index: tools/codechecker/libcodechecker/analyzers/config_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/config_handler.py @@ -0,0 +1,188 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import collections + +from abc import ABCMeta, abstractmethod + +from libcodechecker import logger + +LOG = logger.get_new_logger('CONFIG HANDLER') + +class AnalyzerConfigHandler(object): + """ + handle the checker configurations + and enabled disabled checkers lists + """ + __metaclass__ = ABCMeta + + def __init__(self): + + self.__analyzer_binary = None + self.__analyzer_plugins_dir = None + self.__compiler_sysroot = None + self.__compiler_resource_dirs = [] + self.__sys_inc = [] + self.__includes = [] + self.__analyzer_extra_arguments = '' + + # the key is the checker name, the value is a tuple + # False if disabled (should be by default) + # True if checker is enabled + # (False/True, 'checker_description') + self.__available_checkers = collections.OrderedDict() + + @property + def analyzer_plugins_dir(self): + """ + get directory from where shared objects with checkers should be loaded + """ + return self.__analyzer_plugins_dir + + @analyzer_plugins_dir.setter + def analyzer_plugins_dir(self, value): + """ + set the directory where shared objects with checkers can be found + """ + self.__analyzer_plugins_dir = value + + @property + def analyzer_plugins(self): + """ + full path of the analyzer plugins + """ + plugin_dir = self.__analyzer_plugins_dir + analyzer_plugins = [os.path.join(plugin_dir, f) + for f in os.listdir(plugin_dir) + if os.path.isfile(os.path.join(plugin_dir, f))] + return analyzer_plugins + + @property + def analyzer_binary(self): + return self.__analyzer_binary + + @analyzer_binary.setter + def analyzer_binary(self, value): + self.__analyzer_binary = value + + @abstractmethod + def get_checker_configs(self): + """ + return a lis of (checker_name, key, key_valye) tuples + """ + pass + + def add_checker(self, checker_name, enabled, description): + """ + add additional checker + tuple of (checker_name, True\False) + """ + self.__available_checkers[checker_name] = (enabled, description) + + def enable_checker(self, checker_name, description=None): + """ + enable checker, keep description if already set + """ + for ch_name, values in self.__available_checkers.iteritems(): + if ch_name.startswith(checker_name): + _, description = values + self.__available_checkers[ch_name] = (True, description) + + def disable_checker(self, checker_name, description=None): + """ + disable checker, keep description if already set + """ + for ch_name, values in self.__available_checkers.iteritems(): + if ch_name.startswith(checker_name): + _, description = values + self.__available_checkers[ch_name] = (False, description) + + def checks(self): + """ + return the checkers + """ + return self.__available_checkers + + @property + def compiler_sysroot(self): + """ + get compiler sysroot + """ + return self.__compiler_sysroot + + @compiler_sysroot.setter + def compiler_sysroot(self, compiler_sysroot): + """ + set compiler sysroot + """ + self.__compiler_sysroot = compiler_sysroot + + @property + def compiler_resource_dirs(self): + """ + set compiler resource directories + """ + return self.__compiler_resource_dirs + + @compiler_resource_dirs.setter + def compiler_resource_dirs(self, resource_dirs): + """ + set compiler resource directories + """ + self.__compiler_resource_dirs = resource_dirs + + @property + def system_includes(self): + """ + """ + return self.__sys_inc + + @system_includes.setter + def system_includes(self, includes): + """ + """ + self.__sys_inc = includes + + def add_system_includes(self, sys_inc): + """ + add additional system includes if needed + """ + self.__sys_inc.append(sys_inc) + + @property + def includes(self): + """ + add additional includes if needed + """ + return self.__includes + + @includes.setter + def includes(self, includes): + """ + add additional includes if needed + """ + self.__includes = includes + + def add_includes(self, inc): + """ + add additional include paths + """ + self.__includes.append(inc) + + @property + def analyzer_extra_arguments(self): + """ + extra arguments fowarded to the analyzer without modification + """ + return self.__analyzer_extra_arguments + + @analyzer_extra_arguments.setter + def analyzer_extra_arguments(self, value): + """ + extra arguments fowarded to the analyzer without modification + """ + self.__analyzer_extra_arguments = value Index: tools/codechecker/libcodechecker/analyzers/config_handler_clang_tidy.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/config_handler_clang_tidy.py @@ -0,0 +1,114 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import re +import json +import shlex +import argparse + +from libcodechecker.analyzers import config_handler + +from libcodechecker import logger + +LOG = logger.get_new_logger('CLANG TIDY CONFIG') + +class ClangTidyConfigHandler(config_handler.AnalyzerConfigHandler): + ''' + Configuration handler for Clang-tidy analyzer + ''' + + def __init__(self): + super(ClangTidyConfigHandler, self).__init__() + + def get_checker_configs(self): + """ + process the raw extra analyzer arguments and get the configuration + data ('-config=' argument for Clang tidy) for the checkers + + Clang tidy accepts YAML or JSON formatted config, right now + parsing only the JSON format is supported + + return a lis of tuples + (checker_name, key, key_value) list + + { + "CheckOptions": [ + { + "key": "readability-simplify-boolean-expr.ChainedConditionalReturn", + "value": 1 + }, + { + "key": "google-readability-namespace-comments.SpacesBeforeComments", + "value": 2 + }, + { + "key": "modernize-loop-convert.NamingStyle", + "value": "UPPER_CASE" + }, + { + "key": "clang-analyzer-unix.Malloc:Optimistic", + "value": true + }, + { + "key": "clang-analyzer-unix.test.Checker:Optimistic", + "value": true + } + ] + } + """ + LOG.debug(self.analyzer_extra_arguments) + + res = [] + + # match for clang static analyzer names and attributes + clang_sa_checker_rex = r'^clang-analyzer-(?P([^:]+))\:(?P([^:]+))$' + + # match for clang tidy analyzer names and attributes + clang_tidy_checker_rex = r'^(?P([^.]+))\.(?P([^.]+))$' + + clangsa_pattern = re.compile(clang_sa_checker_rex) + tidy_pattern = re.compile(clang_tidy_checker_rex) + + # get config from the extra arguments if there is any + try: + tidy_config_parser = argparse.ArgumentParser() + tidy_config_parser.add_argument('-config', + dest='tidy_config', + default='', + type=str) + + args, _ = tidy_config_parser.parse_known_args( + shlex.split(self.analyzer_extra_arguments)) + + except Exception as ex: + LOG.debug('No config found in the tidy extra args.') + LOG.debug(ex) + return res + + try: + tidy_config = json.loads(args.tidy_config) + for checker_option in tidy_config.get('CheckOptions', []): + value = checker_option['value'] + key_values = re.match(clangsa_pattern, checker_option['key']) + key_values_tidy = re.match(tidy_pattern, checker_option['key']) + if key_values: + checker_name = key_values.group('checker_name') + checker_attr = key_values.group('checker_attribute') + res.append((checker_name, checker_attr, value)) + elif key_values_tidy: + checker_name = key_values_tidy.group('checker_name') + checker_attr = key_values_tidy.group('checker_attribute') + res.append((checker_name, checker_attr, value)) + else: + LOG.debug('no match') + except ValueError as verr: + LOG.debug('Failed to parse config') + LOG.debug(verr) + except Exception as ex: + LOG.debug('Failed to process config') + LOG.debug(ex) + + return res Index: tools/codechecker/libcodechecker/analyzers/config_handler_clangsa.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/config_handler_clangsa.py @@ -0,0 +1,58 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import re + +from libcodechecker import logger + +from libcodechecker.analyzers import config_handler + +LOG = logger.get_new_logger('CLANGSA CONFIG HANDLER') + +class ClangSAConfigHandler(config_handler.AnalyzerConfigHandler): + """ + Configuration handler for the clang static analyzer + + """ + + def __init__(self): + super(ClangSAConfigHandler, self).__init__() + self.__checker_configs = [] + + def add_checker_config(self, config): + """ + add a (checker_name, key, value) tuple to the list + """ + self.__checker_configs.append(config) + + def get_checker_configs(self): + """ + Process raw config data (from the command line) + Filter out checker related configuration values + like unix:Optimistic=true in the command line: + '-Xanalyzer -analyzer-config -Xanalyzer unix:Optimistic=true' + + return a list of (checker_name, key, value) tuples + """ + + checker_config_pattern = r'(?P([^: ]+))\:(?P([^:=]+))\=(?P([^:\. ]+))' + + raw_config = self.analyzer_extra_arguments + LOG.debug_analyzer(raw_config) + + checker_configs = re.finditer(checker_config_pattern, raw_config) + + if checker_configs: + for cfg in checker_configs: + checker_name = cfg.group('checker_name') + checker_attr = cfg.group('checker_attr') + attr_value = cfg.group('attr_value') + self.__checker_configs.append((checker_name, + checker_attr, + attr_value)) + LOG.debug_analyzer(self.__checker_configs) + + return self.__checker_configs Index: tools/codechecker/libcodechecker/analyzers/result_handler_base.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/result_handler_base.py @@ -0,0 +1,232 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import uuid +import ntpath + +from abc import ABCMeta, abstractmethod + +from libcodechecker import logger + +LOG = logger.get_new_logger('RESULT_HANDLER_BASE') + + +class ResultHandler(object): + """ + handle and store the results at runtime for the analyzer + stdout, stderr, temporarly generated files + do result postprocessing if required + + For each buildaction there is one result handler + the result handler can handle multiple results if there + are more than one analyzed source file in one buildaction + + handle_results() handles the results of a static analysis tool processed on + a build action. Alternatively one can also handle results given by plist + files for which handle_plist() can be used. + + For each build action + - postprocess_result and handle_results can be called multiple times + for the source files in one buildaction they will be analyzed separately + with the same build options + + Method call order should be this + postprocess_result() + handle_results() + + """ + + __metaclass__ = ABCMeta + # handle the output stdout, or plist or both for an analyzer + + def __init__(self, action, workspace): + """ + put the temporary files for the workspace + """ + self.__workspace = workspace + + self.__analyzer_cmd = [] + self.__analyzer_stdout = '' + self.__analyzer_stderr = '' + self.__severity_map = {} + self.__skiplist_handler = None + self.__analyzed_source_file = None + self.__analyzer_returncode = 1 + self.__buildaction = action + + self.__res_file = None + + @property + def buildaction(self): + """ + """ + return self.__buildaction + + @property + def analyzer_cmd(self): + """ + set the analyzer cmd + """ + return self.__analyzer_cmd + + @analyzer_cmd.setter + def analyzer_cmd(self, cmd): + """ + set the analyzer cmd + """ + self.__analyzer_cmd = cmd + + @property + def result_files(self): + """ + return the analyzer result files + """ + return self.__result_files + + @property + def skiplist_handler(self): + """ + """ + return self.__skiplist_handler + + @skiplist_handler.setter + def skiplist_handler(self, handler): + """ + used to check if analyzer result should be + handled or just skipped + """ + self.__skiplist_handler = handler + + @property + def severity_map(self): + """ + severity map for each checker + """ + return self.__severity_map + + @severity_map.setter + def severity_map(self, value): + """ + severity map for each checker + """ + self.__severity_map = value + + @property + def workspace(self): + """ + workspace where the analysis results should go + temporarly generated files + """ + return self.__workspace + + @property + def analyzer_returncode(self): + """ + """ + return self.__analyzer_returncode + + @analyzer_returncode.setter + def analyzer_returncode(self, return_code): + """ + """ + self.__analyzer_returncode = return_code + + @property + def analyzer_stdout(self): + """ + get the stdout from the analyzer + """ + return self.__analyzer_stdout + + @analyzer_stdout.setter + def analyzer_stdout(self, stdout): + """ + set the stdout of the analyzer + """ + self.__analyzer_stdout = stdout + + @property + def analyzer_stderr(self): + """ + get stderr of the analyzer + """ + return self.__analyzer_stderr + + @analyzer_stderr.setter + def analyzer_stderr(self, stderr): + """ + set the stderr of the analyzer + """ + self.__analyzer_stderr = stderr + + @property + def analyzed_source_file(self): + """ + the source file which is analyzed + """ + return self.__analyzed_source_file + + @analyzed_source_file.setter + def analyzed_source_file(self, file_path): + """ + the source file which is analyzed + """ + self.__analyzed_source_file = file_path + + def get_analyzer_result_file(self): + """ + generate a result filename where the analyzer should put the analyzer result + result file should be removed by the result handler if not required anymore + """ + if not self.__res_file: + analyzed_file = self.analyzed_source_file + _, analyzed_file_name = ntpath.split(analyzed_file) + + uid = str(uuid.uuid1()).split('-')[0] + + out_file_name = str(self.buildaction.analyzer_type) + \ + '_' + analyzed_file_name + '_' + uid + '.plist' + + out_file = os.path.join(self.__workspace, out_file_name) + self.__res_file = out_file + + return self.__res_file + + def clean_results(self): + """ + should be called after the postprocessing and result handling is done + """ + if self.__res_file: + try: + os.remove(self.__res_file) + except OSError as oserr: + # there might be no result file if analysis failed + LOG.debug(oserr) + pass + + + @abstractmethod + def postprocess_result(self): + """ + postprocess result if needed + should be called after the analisys finished + """ + pass + + @abstractmethod + def handle_plist(self, plist): + """ + handle the results directly from the given plist file + """ + pass + + @abstractmethod + def handle_results(self): + """ + handle the results + """ + pass Index: tools/codechecker/libcodechecker/analyzers/result_handler_clang_tidy.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/result_handler_clang_tidy.py @@ -0,0 +1,59 @@ +# ------------------------------------------------------------------------- +# The CodeChecker Infrastructure +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# ------------------------------------------------------------------------- + +from libcodechecker import logger +from libcodechecker import tidy_output_converter + +from libcodechecker.analyzers.result_handler_plist_to_db import PlistToDB +from libcodechecker.analyzers.result_handler_plist_to_stdout import PlistToStdout + +LOG = logger.get_new_logger('CLANG_TIDY_RESULT_HANDLER') + + +def generate_plist_from_tidy_result(output_file, tidy_stdout): + """ + Generate a plist file from the clang tidy analyzer results + """ + parser = tidy_output_converter.OutputParser() + + messages = parser.parse_messages(tidy_stdout) + + plist_converter = tidy_output_converter.PListConverter() + plist_converter.add_messages(messages) + + plist_converter.write_to_file(output_file) + + +class ClangTidyPlistToDB(PlistToDB): + """ + Store clang tidy plist results to a database + """ + + def postprocess_result(self): + """ + Generate plist file which can be parsed and processed for + results which can be stored into the database + """ + output_file = self.get_analyzer_result_file() + LOG.debug_analyzer(self.analyzer_stdout) + tidy_stdout = self.analyzer_stdout.splitlines() + generate_plist_from_tidy_result(output_file, tidy_stdout) + + +class ClangTidyPlistToStdout(PlistToStdout): + """ + Print the clang tidy results to the standatd output + """ + + def postprocess_result(self): + """ + For the same output format with the clang static analyzer the + Clang tidy results are postprocessed into a plist file + """ + + output_file = self.get_analyzer_result_file() + tidy_stdout = self.analyzer_stdout.splitlines() + generate_plist_from_tidy_result(output_file, tidy_stdout) Index: tools/codechecker/libcodechecker/analyzers/result_handler_clangsa.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/result_handler_clangsa.py @@ -0,0 +1,36 @@ +# ------------------------------------------------------------------------- +# The CodeChecker Infrastructure +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# ------------------------------------------------------------------------- +""" +result handlers for the Clang Static Analyzer +""" + +from libcodechecker.analyzers.result_handler_plist_to_db import PlistToDB +from libcodechecker.analyzers.result_handler_plist_to_stdout import PlistToStdout + + +class ClangSAPlistToDB(PlistToDB): + """ + Store the result plist generated by the Clang Static Analyzer + to the database + """ + + def postprocess_result(self): + """ + No postprocessing required for clang static analyzer + """ + pass + + +class ClangSAPlistToStdout(PlistToStdout): + """ + Print the results from the generated plist file to the standard output + """ + + def postprocess_result(self): + """ + No postprocessing required for clang static analyzer + """ + pass Index: tools/codechecker/libcodechecker/analyzers/result_handler_plist_to_db.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/result_handler_plist_to_db.py @@ -0,0 +1,218 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import zlib +import ntpath + +from abc import ABCMeta + +from libcodechecker import client +from libcodechecker import logger +from libcodechecker import plist_parser +from libcodechecker import suppress_handler + +from libcodechecker.analyzers.result_handler_base import ResultHandler + +import shared + +LOG = logger.get_new_logger('PLIST TO DB') + + +class PlistToDB(ResultHandler): + """ + Result handler for processing a plist file with the + analysis results and stores them to the database. + """ + + __metaclass__ = ABCMeta + + def __init__(self, buildaction, workspace, run_id): + super(PlistToDB, self).__init__(buildaction, workspace) + self.__run_id = run_id + + def __store_bugs(self, files, bugs, connection, analisys_id): + file_ids = {} + # Send content of file to the server if needed + for file_name in files: + file_descriptor = connection.need_file_content(self.__run_id, + file_name) + file_ids[file_name] = file_descriptor.fileId + + # sometimes the file doesn't exist, e.g. when the input of the + # analysis is pure plist files + if not os.path.isfile(file_name): + LOG.debug(file_name + ' not found, and will not be stored') + continue + + if file_descriptor.needed: + with open(file_name, 'r') as source_file: + file_content = source_file.read() + compressed_file = zlib.compress(file_content, + zlib.Z_BEST_COMPRESSION) + # TODO: we may not use the file content in the end + # depending on skippaths + LOG.debug('storing file content to the database') + connection.add_file_content(file_descriptor.fileId, + compressed_file) + + # skipping bugs in header files handled here + report_ids = [] + for bug in bugs: + events = bug.events() + + # skip list handler can be None if no config file is set + if self.skiplist_handler: + if events and self.skiplist_handler.should_skip( + events[-1].start_pos.file_path): + # Issue #20: this bug is in a file which should be skipped + LOG.debug(bug.hash_value + ' is skipped (in ' + + events[-1].start_pos.file_path + ")") + continue + + # create remaining data for bugs and send them to the server + bug_paths = [] + for path in bug.paths(): + bug_paths.append( + shared.ttypes.BugPathPos(path.start_pos.line, + path.start_pos.col, + path.end_pos.line, + path.end_pos.col, + file_ids[path.start_pos.file_path])) + + bug_events = [] + for event in bug.events(): + bug_events.append(shared.ttypes.BugPathEvent( + event.start_pos.line, + event.start_pos.col, + event.end_pos.line, + event.end_pos.col, + event.msg, + file_ids[event.start_pos.file_path])) + + bug_hash = bug.hash_value + + severity_name = self.severity_map.get(bug.checker_name, + 'UNSPECIFIED') + severity = shared.ttypes.Severity._NAMES_TO_VALUES[severity_name] + + suppress = False + + source_file = bug.file_path + last_bug_event = bug.events()[-1] + bug_line = last_bug_event.start_pos.line + + sp_handler = suppress_handler.SourceSuppressHandler(source_file, + bug_line) + # check for suppress comment + supp = sp_handler.check_source_suppress() + if supp: + # something shoud be suppressed + suppress_checkers = sp_handler.suppressed_checkers() + + if bug.checker_name in suppress_checkers or \ + suppress_checkers == ['all']: + suppress = True + + file_path, file_name = ntpath.split(source_file) + + # checker_hash, file_name, comment + to_suppress = (bug_hash, + file_name, + sp_handler.suppress_comment()) + + LOG.debug(to_suppress) + + connection.add_suppress_bug(self.__run_id, [to_suppress]) + + LOG.debug('Storing check results to the database') + + report_id = connection.add_report(analisys_id, + file_ids[bug.file_path], + bug_hash, + bug.msg, + bug_paths, + bug_events, + bug.checker_name, + bug.category, + bug.type, + severity, + suppress) + + report_ids.append(report_id) + + def handle_plist(self, plist): + with client.get_connection() as connection: + # TODO: When the analyzer name can be read from PList, then it + # should be passed too. + # TODO: File name should be read from the PList and passed. + analisys_id = connection.add_build_action(self.__run_id, + plist, + 'Build action from plist', + '', + '') + + try: + files, bugs = plist_parser.parse_plist(plist) + except Exception as ex: + msg = 'Parsing the generated result file failed' + LOG.error(msg + ' ' + plist) + connection.finish_build_action(analysis_id, msg) + return 1 + + self.__store_bugs(files, bugs, connection, analisys_id) + + connection.finish_build_action(analisys_id, self.analyzer_stderr) + + def handle_results(self): + """ + send the plist content to the database + server API calls should be used in one connection + - addBuildAction + - addReport + - needFileContent + - addFileContent + - finishBuildAction + """ + + with client.get_connection() as connection: + + LOG.debug('Storing original build and analyzer command to the database') + + _, source_file_name = ntpath.split(self.analyzed_source_file) + + analisys_id = connection.add_build_action(self.__run_id, + self.buildaction.original_command, + ' '.join(self.analyzer_cmd), + self.buildaction.analyzer_type, + source_file_name) + + # store buildaction and analyzer command to the database + + + if self.analyzer_returncode == 0: + + LOG.info(self.buildaction.analyzer_type + ' analyzed ' + + source_file_name + ' successfully.') + + plist_file = self.get_analyzer_result_file() + + try: + files, bugs = plist_parser.parse_plist(plist_file) + except Exception as ex: + LOG.debug(str(ex)) + msg = 'Parsing the generated result file failed' + LOG.error(msg + ' ' + plist_file) + connection.finish_build_action(analisys_id, msg) + return 1 + + self.__store_bugs(files, bugs, connection, analisys_id) + else: + LOG.info('Analysing ' + source_file_name + + ' with ' + self.buildaction.analyzer_type + + ' failed.') + + connection.finish_build_action(analisys_id, self.analyzer_stderr) Index: tools/codechecker/libcodechecker/analyzers/result_handler_plist_to_stdout.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/analyzers/result_handler_plist_to_stdout.py @@ -0,0 +1,128 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import sys +import math +import ntpath +import linecache + +from abc import ABCMeta + +from libcodechecker import logger +from libcodechecker import plist_parser + +from libcodechecker.analyzers.result_handler_base import ResultHandler + +LOG = logger.get_new_logger('PLIST TO STDOUT') + + +class PlistToStdout(ResultHandler): + """ + Result handler for processing a plist file with the + analysis results and print them to the standard output. + """ + + __metaclass__ = ABCMeta + + def __init__(self, buildaction, workspace): + super(PlistToStdout, self).__init__(buildaction, workspace) + self.__print_steps = False + self.__output = sys.stdout + + @property + def print_steps(self): + """ + Print the multiple steps for a bug if there is any + """ + return self.__print_steps + + @print_steps.setter + def print_steps(self, value): + """ + Print the multiple steps for a bug if there is any + """ + self.__print_steps = value + + def __format_location(self, event): + pos = event.start_pos + line = linecache.getline(pos.file_path, pos.line) + if line == '': + return line + + marker_line = line[0:(pos.col-1)] + marker_line = ' ' * (len(marker_line) + marker_line.count('\t')) + return '%s%s^' % (line.replace('\t', ' '), marker_line) + + def __format_bug_event(self, event): + pos = event.start_pos + fname = os.path.basename(pos.file_path) + return '%s:%d:%d: %s' % (fname, pos.line, pos.col, event.msg) + + def __print_bugs(self, bugs): + + index_format = ' %%%dd, ' % int(math.floor(math.log10(len(bugs)))+1) + + for bug in bugs: + last_event = bug.get_last_event() + self.__output.write(self.__format_bug_event(last_event)) + self.__output.write('\n') + self.__output.write(self.__format_location(last_event)) + self.__output.write('\n') + if self.__print_steps: + self.__output.write(' Steps:\n') + for index, event in enumerate(bug.events()): + self.__output.write(index_format % (index + 1)) + self.__output.write(self.__format_bug_event(event)) + self.__output.write('\n') + self.__output.write('\n') + + def handle_plist(self, plist): + try: + _, bugs = plist_parser.parse_plist(plist) + except Exception as ex: + LOG.error('The generated plist is not valid!') + LOG.error(ex) + return 1 + + if len(bugs) > 0: + self.__output.write("%s contains %d defect(s)\n\n" % (plist, len(bugs))) + self.__print_bugs(bugs) + else: + self.__output.write("%s doesn't contain any defects\n" % plist) + + def handle_results(self): + + source = self.analyzed_source_file + _, source_file_name = ntpath.split(source) + plist = self.get_analyzer_result_file() + + try: + _, bugs = plist_parser.parse_plist(plist) + except Exception as ex: + LOG.error('The generated plist is not valid!') + LOG.error(ex) + return 1 + + err_code = self.analyzer_returncode + + if err_code == 0: + + if len(bugs) > 0: + self.__output.write('%s found %d defect(s) while analyzing %s\n\n' % + (self.buildaction.analyzer_type, len(bugs), + source_file_name)) + self.__print_bugs(bugs) + else: + self.__output.write('%s found no defects while analyzing %s\n' % + (self.buildaction.analyzer_type, + source_file_name)) + return err_code + else: + self.__output.write('Analyzing %s with %s failed.\n' % + (source_file_name, + self.buildaction.analyzer_type)) + return err_code Index: tools/codechecker/libcodechecker/arg_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/arg_handler.py @@ -0,0 +1,465 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +handle command line arguments +''' +import json +import multiprocessing +import os +import shutil +import sys +import tempfile + +from libcodechecker import analyzer +from libcodechecker import analyzer_env +from libcodechecker import build_action +from libcodechecker import build_manager +from libcodechecker import client +from libcodechecker import debug_reporter +from libcodechecker import generic_package_context +from libcodechecker import generic_package_suppress_handler +from libcodechecker import host_check +from libcodechecker import log_parser +from libcodechecker import logger +from libcodechecker import util +from libcodechecker.analyzers import analyzer_types +from libcodechecker.database_handler import SQLServer +from libcodechecker.viewer_server import client_db_access_server + +LOG = logger.get_new_logger('ARG_HANDLER') + + +def handle_list_checkers(args): + """ + list the supported checkers by the analyzers + list the default enabled and disabled checkers + in the config + """ + context = generic_package_context.get_context() + enabled_analyzers = args.analyzers + analyzer_environment = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + if not enabled_analyzers: + # noting set list checkers for all supported analyzers + enabled_analyzers = list(analyzer_types.supported_analyzers) + + enabled_analyzer_types = set() + for ea in enabled_analyzers: + if ea not in analyzer_types.supported_analyzers: + LOG.info('Not supported analyzer ' + str(ea)) + sys.exit(1) + else: + enabled_analyzer_types.add(ea) + + analyzer_config_map = analyzer_types.build_config_handlers(args, + context, + enabled_analyzer_types) + + for ea in enabled_analyzers: + # get the config + config_handler = analyzer_config_map.get(ea) + source_analyzer = analyzer_types.construct_analyzer_type(ea, + config_handler, + None) + + checkers = source_analyzer.get_analyzer_checkers(config_handler, + analyzer_environment) + + default_checker_cfg = context.default_checkers_config.get(ea+'_checkers') + + analyzer_types.initialize_checkers(config_handler, + checkers, + default_checker_cfg) + for checker_name, value in config_handler.checks().iteritems(): + enabled, description = value + if enabled: + print(' + {0:50} {1}'.format(checker_name, description)) + else: + print(' - {0:50} {1}'.format(checker_name, description)) + +def handle_server(args): + """ + starts the report viewer server + """ + if not host_check.check_zlib(): + LOG.error("zlib error") + sys.exit(1) + + try: + workspace = args.workspace + except AttributeError: + # if no workspace value was set for some reason + # in args set the default value + workspace = util.get_default_workspace() + + # WARNING + # in case of SQLite args.dbaddress default value is used + # for which the is_localhost should return true + + local_db = util.is_localhost(args.dbaddress) + if local_db and not os.path.exists(workspace): + os.makedirs(workspace) + + if args.suppress is None: + LOG.warning('WARNING! No suppress file was given, suppressed results will be only stored in the database.') + + else: + if not os.path.exists(args.suppress): + LOG.error('Suppress file '+args.suppress+' not found!') + sys.exit(1) + + context = generic_package_context.get_context() + context.codechecker_workspace = workspace + context.db_username = args.dbusername + + check_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + sql_server = SQLServer.from_cmdline_args(args, + context.codechecker_workspace, + context.migration_root, + check_env) + conn_mgr = client.ConnectionManager(sql_server, args.check_address, args.check_port) + if args.check_port: + LOG.debug('Starting codechecker server and database server.') + sql_server.start(context.db_version_info, wait_for_start=True, init=True) + conn_mgr.start_report_server(context.db_version_info) + else: + LOG.debug('Starting database.') + sql_server.start(context.db_version_info, wait_for_start=True, init=True) + + # start database viewer + db_connection_string = sql_server.get_connection_string() + + suppress_handler = generic_package_suppress_handler.GenericSuppressHandler() + try: + suppress_handler.suppress_file = args.suppress + LOG.debug('Using suppress file: ' + str(suppress_handler.suppress_file)) + except AttributeError as aerr: + # suppress file was not set + LOG.debug(aerr) + + package_data = {} + package_data['www_root'] = context.www_root + package_data['doc_root'] = context.doc_root + + checker_md_docs = os.path.join(context.doc_root, 'checker_md_docs') + + checker_md_docs_map = os.path.join(checker_md_docs, + 'checker_doc_map.json') + + package_data['checker_md_docs'] = checker_md_docs + + with open(checker_md_docs_map, 'r') as dFile: + checker_md_docs_map = json.load(dFile) + + package_data['checker_md_docs_map'] = checker_md_docs_map + + client_db_access_server.start_server(package_data, + args.view_port, + db_connection_string, + suppress_handler, + args.not_host_only, + context.db_version_info) + + +def handle_log(args): + """ + Generates a build log by running the original build command + No analysis is done + """ + args.logfile = os.path.realpath(args.logfile) + if os.path.exists(args.logfile): + os.remove(args.logfile) + + context = generic_package_context.get_context() + open(args.logfile, 'a').close() # same as linux's touch + build_manager.perform_build_command(args.logfile, + args.command, + context) + + +def handle_debug(args): + """ + Runs a debug command on the buildactions where the analysis + failed for some reason + """ + context = generic_package_context.get_context() + + try: + workspace = args.workspace + except AttributeError: + # if no workspace value was set for some reason + # in args set the default value + workspace = util.get_default_workspace() + + context.codechecker_workspace = workspace + context.db_username = args.dbusername + + check_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + sql_server = SQLServer.from_cmdline_args(args, + context.codechecker_workspace, + context.migration_root, + check_env) + sql_server.start(context.db_version_info, wait_for_start=True, init=False) + + debug_reporter.debug(context, sql_server.get_connection_string(), args.force) + + +def handle_check(args): + """ + Runs the original build and logs the buildactions + Based on the log runs the analysis + """ + try: + + if not host_check.check_zlib(): + LOG.error("zlib error") + sys.exit(1) + + try: + workspace = args.workspace + except AttributeError: + # if no workspace value was set for some reason + # in args set the default value + workspace = util.get_default_workspace() + + workspace = os.path.realpath(workspace) + if not os.path.isdir(workspace): + os.mkdir(workspace) + + context = generic_package_context.get_context() + context.codechecker_workspace = workspace + context.db_username = args.dbusername + + log_file = build_manager.check_log_file(args) + + if not log_file: + log_file = build_manager.generate_log_file(args, + context) + if not log_file: + LOG.error("Failed to generate compilation command file: " + + log_file) + sys.exit(1) + + try: + actions = log_parser.parse_log(log_file) + except Exception as ex: + LOG.error(ex) + sys.exit(1) + + if not actions: + LOG.warning('There are no build actions in the log file.') + sys.exit(1) + + check_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + sql_server = SQLServer.from_cmdline_args(args, + context.codechecker_workspace, + context.migration_root, + check_env) + + conn_mgr = client.ConnectionManager(sql_server, 'localhost', util.get_free_port()) + + sql_server.start(context.db_version_info, wait_for_start=True, init=True) + + conn_mgr.start_report_server(context.db_version_info) + + LOG.debug("Checker server started.") + + analyzer.run_check(args, + actions, + context) + + LOG.info("Analysis has finished.") + + db_data = "" + if args.postgresql: + db_data += " --postgresql" \ + + " --dbname " + args.dbname \ + + " --dbport " + str(args.dbport) \ + + " --dbusername " + args.dbusername + + LOG.info("To view results run:\nCodeChecker server -w " + + workspace + db_data) + + except Exception as ex: + LOG.error(ex) + import traceback + print(traceback.format_exc()) + + +def _do_quickcheck(args): + ''' + Handles the "quickcheck" command. + + For arguments see main function in CodeChecker.py. It also requires an extra + property in args object, namely workspace which is a directory path as a + string. This function is called from handle_quickcheck. + ''' + + context = generic_package_context.get_context() + + try: + workspace = args.workspace + except AttributeError: + # if no workspace value was set for some reason + # in args set the default value + workspace = util.get_default_workspace() + + context.codechecker_workspace = workspace + + # load severity map from config file + if os.path.exists(context.checkers_severity_map_file): + with open(context.checkers_severity_map_file, 'r') as sev_conf_file: + severity_config = sev_conf_file.read() + + context.severity_map = json.loads(severity_config) + + log_file = build_manager.check_log_file(args) + + if not log_file: + log_file = build_manager.generate_log_file(args, + context) + if not log_file: + LOG.error("Failed to generate compilation command file: " + log_file) + sys.exit(1) + + try: + actions = log_parser.parse_log(log_file) + except Exception as ex: + LOG.error(ex) + sys.exit(1) + + if not actions: + LOG.warning('There are no build actions in the log file.') + sys.exit(1) + + analyzer.run_quick_check(args, context, actions) + + +def handle_quickcheck(args): + ''' + Handles the "quickcheck" command using _do_quickcheck function. + + It creates a new temporary directory and sets it as workspace directory. + After _do_quickcheck call it deletes the temporary directory and its + content. + ''' + + args.workspace = tempfile.mkdtemp(prefix='codechecker-qc') + try: + _do_quickcheck(args) + finally: + shutil.rmtree(args.workspace) + + +def consume_plist(item): + plist, args, context = item + LOG.info('Consuming ' + plist) + + action = build_action.BuildAction() + action.analyzer_type = analyzer_types.CLANG_SA + action.original_command = 'Imported from PList directly' + + rh = analyzer_types.construct_result_handler(args, + action, + context.run_id, + args.directory, + {}, + None, + not args.stdout) + + rh.handle_plist(os.path.join(args.directory, plist)) + +def handle_plist(args): + + context = generic_package_context.get_context() + context.codechecker_workspace = args.workspace + context.db_username = args.dbusername + + if not args.stdout: + args.workspace = os.path.realpath(args.workspace) + if not os.path.isdir(args.workspace): + os.mkdir(args.workspace) + + check_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + sql_server = SQLServer.from_cmdline_args(args, + context.codechecker_workspace, + context.migration_root, + check_env) + + conn_mgr = client.ConnectionManager(sql_server, + 'localhost', + util.get_free_port()) + + sql_server.start(context.db_version_info, wait_for_start=True, init=True) + + conn_mgr.start_report_server(context.db_version_info) + + with client.get_connection() as connection: + package_version = context.version['major'] + '.' + context.version['minor'] + context.run_id = connection.add_checker_run(' '.join(sys.argv), + args.name, + package_version, + args.force) + + pool = multiprocessing.Pool(args.jobs) + + try: + items = [(plist, args, context) for plist in os.listdir(args.directory)] + pool.map_async(consume_plist, items, 1).get(float('inf')) + pool.close() + except Exception: + pool.terminate() + raise + finally: + pool.join() + +def handle_version_info(args): + ''' Get and print the version informations from the + version config file and thrift API versions''' + + context = generic_package_context.get_context() + version_file = context.version_file + + v_data = '' + try: + with open(version_file) as v_file: + v_data = v_file.read() + + version_data = json.loads(v_data) + base_version = version_data['version']['major'] + \ + '.' + version_data['version']['minor'] + db_schema_version = version_data['db_version']['major'] + \ + '.'+version_data['db_version']['minor'] + + print('Base package version: \t' + base_version).expandtabs(30) + print('Package build date: \t' + + version_data['package_build_date']).expandtabs(30) + print('Git hash: \t' + version_data['git_hash']).expandtabs(30) + print('DB schema version: \t' + db_schema_version).expandtabs(30) + + except ValueError as verr: + LOG.error('Failed to decode version information from the config file.') + LOG.error(verr) + sys.exit(1) + + except IOError as ioerr: + LOG.error('Failed to read version config file: ' + version_file) + LOG.error(ioerr) + sys.exit(1) + + # thift api version for the clients + from codeCheckerDBAccess import constants + print('Thrift client api version: \t' + + constants.API_VERSION).expandtabs(30) Index: tools/codechecker/libcodechecker/build_action.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/build_action.py @@ -0,0 +1,147 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +'''''' + +import hashlib + + +class BuildAction(object): + def __init__(self, build_action_id=0): + self._id = build_action_id + self._analyzer_options = [] + self.analyzer_type = -1 + self._original_command = '' + self._directory = '' + self._output = '' + self._lang = None + self._target = '' + self._source_count = 0 + self._sources = [] + self._skip = True + + def __str__(self): + # for debugging + return ('Id: {0} ,\nOriginal command: {1},\n' + 'Analyzer type: {2},\n Analyzer options: {3},\n' + 'Directory: {4},\nOutput: {5},\nLang: {6},\nTarget: {7},\n' + 'Source count {8},\nSources: {9}').\ + format(self._id, self._original_command, + self._analyzer_type, self._analyzer_options, + self._directory, self._output, self._lang, self._target, + self._source_count, self._sources) + + @property + def analyzer_type(self): + """ + stores which type of analyzer should be run in this + buildaction + """ + return self._analyzer_type + + @analyzer_type.setter + def analyzer_type(self, value): + """ + stores which type of analyzer should be run in this + buildaction + """ + self._analyzer_type = value + + @property + def id(self): + return self._id + + @property + def analyzer_options(self): + return self._analyzer_options + + @analyzer_options.setter + def analyzer_options(self, value): + '''A filtered compile arguments which will be forwarded to the analyzer.''' + self._analyzer_options = value + + @property + def original_command(self): + return self._original_command + + @original_command.setter + def original_command(self, value): + self._original_command = value + + @property + def directory(self): + return self._directory + + @directory.setter + def directory(self, value): + self._directory = value + + @property + def output(self): + return self._output + + @output.setter + def output(self, value): + self._output = value + + @property + def source_count(self): + return self._source_count + + @source_count.setter + def source_count(self, count): + self._source_count = count + + @property + def sources(self): + for source in self._sources: + yield source + + @sources.setter + def sources(self, value): + self._sources.append(value) + self._source_count += 1 + + @property + def lang(self): + return self._lang + + @lang.setter + def lang(self, value): + self._lang = value + + @property + def target(self): + return self._target + + @target.setter + def target(self, value): + self._target = value + + @property + def skip(self): + return self._skip + + @skip.setter + def skip(self, value): + self._skip = value + + def __eq__(self, other): + return other._original_command == self._original_command + + @property + def cmp_key(self): + ''' + If the compilation database contains the same compilation action + multiple times it should be checked only once. + Use this key to compare compilation commands for the analysis. + ''' + hash_content = [] + hash_content.extend(self.analyzer_options) + hash_content.append(str(self.analyzer_type)) + hash_content.append(self.output) + hash_content.append(self.target) + hash_content.extend(self.sources) + return hashlib.sha1(''.join(hash_content)).hexdigest() Index: tools/codechecker/libcodechecker/build_manager.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/build_manager.py @@ -0,0 +1,180 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +""" +build and log related stuff +""" +import os +import pickle +import subprocess +import sys +import shutil +import platform + +from libcodechecker import logger +from libcodechecker import analyzer_env +from libcodechecker import host_check + +from distutils.spawn import find_executable + +LOG = logger.get_new_logger('BUILD MANAGER') + + +def execute_buildcmd(command, silent=False, env=None, cwd=None): + """ + Execute the the build command and continously write + the output from the process to the standard output. + """ + proc = subprocess.Popen(command, + bufsize=-1, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + cwd=cwd, + shell=True) + + while True: + line = proc.stdout.readline() + if not silent: + sys.stdout.write(line) + if line == '' and proc.poll() is not None: + break + + return proc.returncode + + +def perform_build_command(logfile, command, context, silent=False): + """ + Build the project and create a log file. + """ + if not silent: + LOG.info("Starting build ...") + + try: + original_env_file = os.environ['CODECHECKER_ORIGINAL_BUILD_ENV'] + LOG.debug_analyzer('Loading original build env from: ' + original_env_file) + + with open(original_env_file, 'rb') as env_file: + original_env = pickle.load(env_file) + + except Exception as ex: + LOG.warning(str(ex)) + LOG.warning('Failed to get saved original_env using a current copy for logging') + original_env = os.environ.copy() + + return_code = 0 + + # Run user's commands with intercept + if host_check.check_intercept(original_env): + LOG.debug_analyzer("with intercept ...") + final_command = command + command = "intercept-build " + "--cdb "+ logfile + " " + final_command + log_env = original_env + LOG.debug_analyzer(command) + + # Run user's commands in shell + else: + # TODO better platform detection + if platform.system() == 'Linux': + LOG.debug_analyzer("with ld logger ...") + log_env = analyzer_env.get_log_env(logfile, context, original_env) + if 'CC_LOGGER_GCC_LIKE' not in log_env: + log_env['CC_LOGGER_GCC_LIKE'] = 'gcc:g++:clang:clang++:cc:c++' + else: + LOG.error("Intercept-build is required to run CodeChecker in OS X.") + sys.exit(1) + + LOG.debug_analyzer(log_env) + try: + ret_code = execute_buildcmd(command, silent, log_env) + + if not silent: + if ret_code == 0: + LOG.info("Build finished successfully.") + LOG.debug_analyzer("The logfile is: " + logfile) + else: + LOG.info("Build failed.") + sys.exit(ret_code) + + except Exception as ex: + LOG.error("Calling original build command failed") + LOG.error(str(ex)) + sys.exit(1) + + +def default_compilation_db(workspace_path): + """ + default compilation commands database file in the workspace + """ + compilation_commands = os.path.join(workspace_path, + 'compilation_commands.json') + return compilation_commands + + +def check_log_file(args): + """ + check if the compilation command file was set in the command line + if not check if it is in the workspace directory + """ + log_file = None + try: + if args.logfile: + log_file = os.path.realpath(args.logfile) + else: + # log file could be in the workspace directory + log_file = default_compilation_db(args.workspace) + if not os.path.exists(log_file): + LOG.debug_analyzer("Compilation database file does not exists.") + return None + except AttributeError as ex: + # args.log_file was not set + LOG.debug_analyzer(ex) + LOG.debug_analyzer("Compilation database file was not set in the command line.") + finally: + return log_file + + +def generate_log_file(args, context, silent=False): + """ + Returns a build command log file for check/quickcheck command. + """ + + log_file = None + try: + if args.command: + + intercept_build_executable = find_executable('intercept-build') + + if intercept_build_executable == None: + if platform.system() == 'Linux': + # check if logger bin exists + if not os.path.isfile(context.path_logger_bin): + LOG.error('Logger binary not found! Required for logging.') + sys.exit(1) + + # check if logger lib exists + if not os.path.exists(context.path_logger_lib): + LOG.error('Logger library directory not found! Libs are requires' \ + 'for logging.') + sys.exit(1) + + log_file = default_compilation_db(args.workspace) + if os.path.exists(log_file): + LOG.debug_analyzer("Removing previous compilation command file: " + + log_file) + os.remove(log_file) + + open(log_file, 'a').close() # same as linux's touch + + perform_build_command(log_file, + args.command, + context, + silent=silent) + + except AttributeError as aerr: + LOG.error(aerr) + sys.exit(1) + + return log_file Index: tools/codechecker/libcodechecker/checkers.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/checkers.py @@ -0,0 +1,38 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +from codechecker import util +from codechecker import globalConfig +from codechecker import logger + + +def get_env_checkers_list(env_name): + env_set_checkers = util.get_env_var(env_name) + + LOG = logger.get_new_logger("CHECKERS") + LOG.debug_analyzer('Getting checkers list from environment variable %s' % (env_name)) + + if env_set_checkers: + checkers_list = env_set_checkers.split(':') + LOG.debug_analyzer('Checkers list is -> ' + env_set_checkers) + return sorted(checkers_list) + else: + LOG.debug_analyzer('No checkers list was defined.') + return None + + +def get_enabled_checkers(): + config = globalConfig.GlobalConfig() + env_checkers = get_env_checkers_list(config.envEnableCheckersName) + + return env_checkers + + +def get_disabled_checkers(): + config = globalConfig.GlobalConfig() + env_checkers = get_env_checkers_list(config.envDisableCheckersName) + + return env_checkers Index: tools/codechecker/libcodechecker/client.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/client.py @@ -0,0 +1,243 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import sys +import time +import atexit +import contextlib +import multiprocessing +import os +import codecs + +import shared +from libcodechecker.storage_server import report_server + +from DBThriftAPI import CheckerReport +from DBThriftAPI.ttypes import SuppressBugData + +from thrift import Thrift +from thrift.transport import TSocket +from thrift.transport import TTransport +from thrift.protocol import TBinaryProtocol + +from libcodechecker import logger +from libcodechecker import suppress_file_handler + +LOG = logger.get_new_logger('CLIENT') + + +# ----------------------------------------------------------------------------- +def clean_suppress(connection, run_id): + """ + clean all the suppress information from the database + """ + connection.clean_suppress_data(run_id) + + +# ----------------------------------------------------------------------------- +def send_suppress(run_id, connection, file_name): + """ + collect suppress information from the suppress file to be stored + in the database + """ + suppress_data = [] + if os.path.exists(file_name): + with codecs.open(file_name, 'r', 'UTF-8') as s_file: + suppress_data = suppress_file_handler.get_suppress_data(s_file) + + if len(suppress_data) > 0: + connection.add_suppress_bug(run_id, suppress_data) + +# ----------------------------------------------------------------------------- +def replace_config_in_db(run_id, connection, configs): + configuration_list = [] + for checker_name, key, key_value in configs: + configuration_list.append(shared.ttypes.ConfigValue(checker_name, + key, + str(key_value))) + # store checker configs to the database + connection.replace_config_info(run_id, configuration_list) + +# ----------------------------------------------------------------------------- +@contextlib.contextmanager +def get_connection(): + ''' Automatic Connection handler via ContextManager idiom. + You can use this in with statement.''' + connection = ConnectionManager.instance.create_connection() + + try: + yield connection + finally: + connection.close_connection() + + +# ----------------------------------------------------------------------------- +class Connection(object): + ''' Represent a connection to the server. + In contstructor establish the connection and + you have to call close_connection function to close it. + So, you should set it up before create a connection.''' + + def __init__(self, host, port): + ''' Establish the connection beetwen client and server. ''' + + tries_count = 0 + while True: + try: + self._transport = TSocket.TSocket(host, port) + self._transport = TTransport.TBufferedTransport(self._transport) + self._protocol = TBinaryProtocol.TBinaryProtocol(self._transport) + self._client = CheckerReport.Client(self._protocol) + self._transport.open() + break + + except Thrift.TException as thrift_exc: + if tries_count > 3: + LOG.error('The client cannot establish the connection ' + 'with the server!') + LOG.error('%s' % (thrift_exc.message)) + sys.exit(2) + else: + tries_count += 1 + time.sleep(1) + + def close_connection(self): + ''' Close connection. ''' + self._transport.close() + + def add_checker_run(self, command, name, version, force): + run_id = self._client.addCheckerRun(command, name, version, force) + return run_id + + def finish_checker_run(self, run_id): + ''' bool finishCheckerRun(1: i64 run_id) ''' + self._client.finishCheckerRun(run_id) + + def clean_suppress_data(self, run_id): + """ + clean suppress data + """ + self._client.cleanSuppressData(run_id) + + def add_suppress_bug(self, run_id, suppress_data): + """ + process and send suppress data + which should be sent to the report server + """ + bugs_to_suppress = [] + for checker_hash, file_name, comment in suppress_data: + comment = comment.encode('UTF-8') + suppress_bug = SuppressBugData(checker_hash, + file_name, + comment) + bugs_to_suppress.append(suppress_bug) + + return self._client.addSuppressBug(run_id, + bugs_to_suppress) + + def add_skip_paths(self, run_id, paths): + ''' bool addSkipPath(1: i64 run_id, 2: map paths) ''' + # convert before sending through thrift + converted = {} + for path, comment in paths.items(): + converted[path] = comment.encode('UTF-8') + return self._client.addSkipPath(run_id, converted) + + def replace_config_info(self, run_id, config_list): + ''' bool replaceConfigInfo(1: i64 run_id, 2: list values) ''' + return self._client.replaceConfigInfo(run_id, config_list) + + def add_build_action(self, run_id, build_cmd, check_cmd, analyzer_type, + analyzed_source_file): + ''' i64 addBuildAction(1: i64 run_id, 2: string build_cmd, + 3: string check_cmd, 4: string analyzer_type, 5: string analyzed_source_file) + ''' + return self._client.addBuildAction(run_id, + build_cmd, + check_cmd, + analyzer_type, + analyzed_source_file) + + def finish_build_action(self, action_id, failure): + ''' bool finishBuildAction(1: i64 action_id, 2: string failure) ''' + return self._client.finishBuildAction(action_id, failure) + + def add_report(self, build_action_id, file_id, bug_hash, + checker_message, bugpath, events, checker_id, checker_cat, + bug_type, severity, suppress): + ''' ''' + return self._client.addReport(build_action_id, file_id, bug_hash, + checker_message, bugpath, + events, checker_id, checker_cat, + bug_type, severity, suppress) + + def need_file_content(self, run_id, filepath): + ''' NeedFileResult needFileContent(1: i64 run_id, 2: string filepath) + ''' + return self._client.needFileContent(run_id, filepath) + + def add_file_content(self, file_id, file_content): + ''' bool addFileContent(1: i64 file_id, 2: binary file_content) ''' + return self._client.addFileContent(file_id, file_content) + + +# ----------------------------------------------------------------------------- +class ConnectionManager(object): + ''' + ContextManager class for handling connections. + Store common information for about connection. + Start and stop the server. + ''' + + run_id = None + + def __init__(self, database_server, host, port): + self.database_server = database_server + self.host = host + self.port = port + ConnectionManager.instance = self + + def create_connection(self): + return Connection(self.host, self.port) + + def start_report_server(self, db_version_info): + + is_server_started = multiprocessing.Event() + server = multiprocessing.Process(target=report_server.run_server, + args=( + self.port, + self.database_server.get_connection_string(), + db_version_info, + is_server_started)) + + server.daemon = True + server.start() + + # Wait a bit + counter = 0 + while not is_server_started.is_set() and counter < 4: + LOG.debug('Waiting for checker server to start.') + time.sleep(3) + counter += 1 + + if counter >= 4 or not server.is_alive(): + # last chance to start + if server.exitcode is None: + # it is possible that the database starts really slow + time.sleep(5) + if not is_server_started.is_set(): + LOG.error('Failed to start checker server.') + sys.exit(1) + else: + LOG.error('Failed to start checker server.') + LOG.error('Checker server exit code: ' + + str(server.exitcode)) + sys.exit(1) + + atexit.register(server.terminate) + self.server = server + + LOG.debug('Checker server start sequence done.') Index: tools/codechecker/libcodechecker/cmdline_client/__init__.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/cmdline_client/__init__.py @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. Index: tools/codechecker/libcodechecker/cmdline_client/cmd_line_client.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/cmdline_client/cmd_line_client.py @@ -0,0 +1,378 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import sys +import argparse +import json + +import thrift_helper + +import shared +import codeCheckerDBAccess + +SUPPORTED_VERSION = '4.0' + + +def check_API_version(client): + ''' check if server API is supported by the client''' + version = client.getAPIVersion() + supp_major_version = SUPPORTED_VERSION.split('.')[0] + api_major_version = version.split('.')[0] + + if supp_major_version != api_major_version: + return False + + return True + + +class CmdLineOutputEncoder(json.JSONEncoder): + + def default(self, obj): + d = {} + d.update(obj.__dict__) + return d + + +def setupClient(host, port, uri): + ''' setup the thrift client and check API version''' + + client = thrift_helper.ThriftClientHelper(host, port, uri) + # test if client can work with thrift API getVersion + if not check_API_version(client): + print('Backward incompatible change was in the API.') + print('Please update client. Server version is not supported') + sys.exit(1) + + return client + + +def print_table(lines, separate_head=True): + """Prints a formatted table given a 2 dimensional array""" + # Count the column width + + widths = [] + for line in lines: + for i, size in enumerate([len(x) for x in line]): + while i >= len(widths): + widths.append(0) + if size > widths[i]: + widths[i] = size + + # Generate the format string to pad the columns + print_string = "" + for i, width in enumerate(widths): + print_string += "{" + str(i) + ":" + str(width) + "} | " + if (len(print_string) == 0): + return + print_string = print_string[:-3] + + # Print the actual data + print("-"*(sum(widths)+3*(len(widths)-1))) + for i, line in enumerate(lines): + print(print_string.format(*line)) + if (i == 0 and separate_head): + print("-"*(sum(widths)+3*(len(widths)-1))) + print("-"*(sum(widths)+3*(len(widths)-1))) + print '' + + +def get_run_ids(client): + ''' returns a map for run names and run_ids ''' + + runs = client.getRunData() + + run_data = {} + for run in runs: + run_data[run.name] = (run.runId, run.runDate) + + return run_data + + +def check_run_names(client, check_names): + + run_info = get_run_ids(client) + + if not check_names: + return run_info + + missing_name = False + for name in check_names: + if not run_info.get(name): + print('No check name found: ' + name) + missing_name = True + + if missing_name: + print('Possible check names are:') + for name, info in run_info.items(): + print(name) + sys.exit(1) + + return run_info + + +def handle_list_runs(args): + + client = setupClient(args.host, args.port, '/') + runs = client.getRunData() + + if args.output_format == 'json': + results = [] + for run in runs: + results.append({run.name: run}) + print CmdLineOutputEncoder().encode(results) + + else: + rows = [] + rows.append(('Name', 'ResultCount', 'RunDate')) + for run in runs: + rows.append((run.name, str(run.resultCount), run.runDate)) + + print_table(rows) + + +def handle_list_results(args): + + client = setupClient(args.host, args.port, '/') + + run_info = check_run_names(client, [args.name]) + + # for name, info in run_info.items() + run_id, run_date = run_info.get(args.name) + + limit = 500 + offset = 0 + + filters = [] + if args.suppressed: + report_filter = codeCheckerDBAccess.ttypes.ReportFilter(suppressed=True) + else: + report_filter = codeCheckerDBAccess.ttypes.ReportFilter(suppressed=False) + + filters.append(report_filter) + + results = client.getRunResults(run_id, limit, offset, None, filters) + + if args.output_format == 'json': + print CmdLineOutputEncoder().encode(results) + else: + rows = [] + if args.suppressed: + rows.append(('File', 'Checker', 'Severity', 'Msg', 'Suppress comment')) + while results: + for res in results: + bug_line = res.lastBugPosition.startLine + checked_file = res.checkedFile+' @ '+str(bug_line) + sev = shared.ttypes.Severity._VALUES_TO_NAMES[res.severity] + rows.append((checked_file, res.checkerId, sev, res.checkerMsg, res.suppressComment)) + + offset += limit + results = client.getRunResults(run_id, limit, offset, None, filters) + + else: + rows.append(('File', 'Checker', 'Severity', 'Msg')) + while results: + for res in results: + bug_line = res.lastBugPosition.startLine + sev = shared.ttypes.Severity._VALUES_TO_NAMES[res.severity] + checked_file = res.checkedFile+' @ '+str(bug_line) + rows.append((checked_file, res.checkerId, sev, res.checkerMsg)) + + offset += limit + results = client.getRunResults(run_id, limit, offset, None, filters) + + print_table(rows) + + +def handle_list_result_types(args): + + client = setupClient(args.host, args.port, '/') + + filters = [] + if args.suppressed: + report_filter = codeCheckerDBAccess.ttypes.ReportFilter(suppressed=True) + else: + report_filter = codeCheckerDBAccess.ttypes.ReportFilter(suppressed=False) + + filters.append(report_filter) + + if args.all_results: + run_info = check_run_names(client, None) + results_collector = [] + for name, run_info in run_info.items(): + run_id, run_date = run_info + results = client.getRunResultTypes(run_id, filters) + if args.output_format == 'json': + results_collector.append({name: results}) + else: + print('Check date: '+run_date) + print('Check name: '+name) + rows = [] + rows.append(('Checker', 'Severity', 'Count')) + for res in results: + sev = shared.ttypes.Severity._VALUES_TO_NAMES[res.severity] + rows.append((res.checkerId, sev, str(res.count))) + + print_table(rows) + + if args.output_format == 'json': + print CmdLineOutputEncoder().encode(results_collector) + else: + run_info = check_run_names(client, args.names) + for name in args.names: + run_id, run_date = run_info.get(name) + + results = client.getRunResultTypes(run_id, filters) + if args.output_format == 'json': + print CmdLineOutputEncoder().encode(results) + else: + print('Check date: '+run_date) + print('Check name: '+name) + rows = [] + rows.append(('Checker', 'Severity', 'Count')) + for res in results: + sev = shared.ttypes.Severity._VALUES_TO_NAMES[res.severity] + rows.append((res.checkerId, sev, str(res.count))) + + print_table(rows) + + +# ------------------------------------------------------------ +def handle_remove_run_results(args): + + client = setupClient(args.host, args.port, '/') + + run_info = check_run_names(client, args.name) + + # FIXME LIST comprehension + run_ids_to_delete = [] + for name, info in run_info.items(): + run_id, run_date = info + if name in args.name: + run_ids_to_delete.append(run_id) + + client.removeRunResults(run_ids_to_delete) + + print('Done.') + + +def handle_diff_results(args): + def printResult(getterFn, baseid, newid, suppr, output_format): + report_filter = [codeCheckerDBAccess.ttypes.ReportFilter(suppressed=suppr)] + rows, sort_type, limit, offset = [], None, 500, 0 + + rows.append(('File', 'Checker', 'Severity', 'Msg')) + results = getterFn(baseid, newid, limit, offset, sort_type, report_filter) + + if output_format == 'json': + print CmdLineOutputEncoder().encode(results) + else: + while results: + for res in results: + bug_line = res.lastBugPosition.startLine + sev = shared.ttypes.Severity._VALUES_TO_NAMES[res.severity] + checked_file = res.checkedFile+' @ '+str(bug_line) + rows.append((checked_file, res.checkerId, sev, res.checkerMsg)) + + offset += limit + results = getterFn(baseid, newid, limit, offset, sort_type, report_filter) + + print_table(rows) + + client = setupClient(args.host, args.port, '/') + run_info = check_run_names(client, [args.basename, args.newname]) + + baseid = run_info[args.basename][0] + newid = run_info[args.newname][0] + + if args.new: + printResult(client.getNewResults, baseid, newid, args.suppressed, args.output_format) + elif args.unresolved: + printResult(client.getUnresolvedResults, baseid, newid, args.suppressed, args.output_format) + elif args.resolved: + printResult(client.getResolvedResults, baseid, newid, args.suppressed, args.output_format) + + +def register_client_command_line(argument_parser): + ''' should be used to extend the already existing arguments + extend the argument parser with extra commands''' + + subparsers = argument_parser.add_subparsers() + + # list runs + listruns_parser = subparsers.add_parser('runs', help='Get the run data.') + listruns_parser.add_argument('--host', type=str, dest="host", default='localhost', + help='Server host.') + listruns_parser.add_argument('-p', '--port', type=str, dest="port", default=11444, + required=True, help='Server port.') + listruns_parser.add_argument('-o', choices=['plaintext', 'json'], default='plaintext', type=str, dest="output_format", help='Output format.') + listruns_parser.set_defaults(func=handle_list_runs) + + # list results + listresults_parser = subparsers.add_parser('results', help='List results.') + listresults_parser.add_argument('--host', type=str, dest="host", default='localhost', + help='Server host.') + listresults_parser.add_argument('-p', '--port', type=str, dest="port", default=11444, + required=True, help='Server port.') + listresults_parser.add_argument('-n', '--name', type=str, dest="name", required=True, + help='Check name.') + listresults_parser.add_argument('-s', '--suppressed', action="store_true", dest="suppressed", help='Suppressed results.') + listresults_parser.add_argument('-o', choices=['plaintext', 'json'], default='plaintext', type=str, dest="output_format", help='Output format.') + listresults_parser.set_defaults(func=handle_list_results) + + # list diffs + diff_parser = subparsers.add_parser('diff', help='Diff two run.') + diff_parser.add_argument('--host', type=str, dest="host", default='localhost', + help='Server host.') + diff_parser.add_argument('-p', '--port', type=str, dest="port", default=11444, + required=True, help='Server port.') + diff_parser.add_argument('-b', '--basename', type=str, dest="basename", required=True, + help='Base name.') + diff_parser.add_argument('-n', '--newname', type=str, dest="newname", required=True, + help='New name.') + diff_parser.add_argument('-s', '--suppressed', action="store_true", dest="suppressed", default=False, + required=False, help='Show suppressed bugs.') + diff_parser.add_argument('-o', choices=['plaintext', 'json'], default='plaintext', type=str, dest="output_format", help='Output format.') + group = diff_parser.add_mutually_exclusive_group(required=True) + group.add_argument('--new', action="store_true", dest="new", help="Show new results.") + group.add_argument('--unresolved', action="store_true", dest="unresolved", help="Show unresolved results.") + group.add_argument('--resolved', action="store_true", dest="resolved", help="Show resolved results.") + diff_parser.set_defaults(func=handle_diff_results) + + # list resulttypes + sum_parser = subparsers.add_parser('sum', help='Sum results.') + sum_parser.add_argument('--host', type=str, dest="host", default='localhost', + help='Server host.') + sum_parser.add_argument('-p', '--port', type=str, dest="port", default=11444, + required=True, help='Server port.') + name_group = sum_parser.add_mutually_exclusive_group(required=True) + name_group.add_argument('-n', '--name', nargs='+', type=str, dest="names", help='Check name.') + name_group.add_argument('-a', '--all', action='store_true', dest="all_results", help='All results.') + + sum_parser.add_argument('-s', '--suppressed', action="store_true", dest="suppressed", help='Suppressed results.') + sum_parser.add_argument('-o', choices=['plaintext', 'json'], default='plaintext', type=str, dest="output_format", help='Output format.') + sum_parser.set_defaults(func=handle_list_result_types) + + # list resulttypes + sum_parser = subparsers.add_parser('del', help='Remove run results.') + sum_parser.add_argument('--host', type=str, dest="host", default='localhost', + help='Server host.') + sum_parser.add_argument('-p', '--port', type=str, dest="port", default=11444, + required=True, help='Server port.') + sum_parser.add_argument('-n', '--name', nargs='+', type=str, dest="name", required=True, help='Server port.') + sum_parser.set_defaults(func=handle_remove_run_results) + + +def main(): + + parser = argparse.ArgumentParser(description='Simple command line client for codechecker.') + + register_client_command_line(parser) + args = parser.parse_args() + args.func(args) + + +if __name__ == "__main__": + main() Index: tools/codechecker/libcodechecker/cmdline_client/thrift_helper.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/cmdline_client/thrift_helper.py @@ -0,0 +1,108 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import sys +#import datetime +import socket + +from thrift.transport import THttpClient +from thrift.protocol import TJSONProtocol + +from codeCheckerDBAccess import codeCheckerDBAccess +import shared + + +class ThriftClientHelper(): + + def __init__(self, host, port, uri): + self.transport = THttpClient.THttpClient(host, port, uri) + self.protocol = TJSONProtocol.TJSONProtocol(self.transport) + self.client = codeCheckerDBAccess.Client(self.protocol) + +# ------------------------------------------------------------ + def ThriftClientCall(function): + #print type(function) + funcName = function.__name__ + def wrapper(self, *args, **kwargs): + #print('['+host+':'+str(port)+'] >>>>> ['+funcName+']') + #before = datetime.datetime.now() + self.transport.open() + func = getattr(self.client,funcName) + try: + res = func(*args, **kwargs) + + except shared.ttypes.RequestFailed as reqfailure: + if reqfailure.error_code == shared.ttypes.ErrorCode.DATABASE: + + print(str(reqfailure.message)) + else: + print('Other error') + print(str(reqfailure)) + + sys.exit(1) + + except socket.error as serr: + errCause = os.strerror(serr.errno) + print(errCause) + print(str(serr)) + sys.exit(1) + + + #after = datetime.datetime.now() + #timediff = after - before + #diff = timediff.microseconds/1000 + #print('['+str(diff)+'ms] <<<<< ['+host+':'+str(port)+']') + #print res + self.transport.close() + return res + + return wrapper + + # ------------------------------------------------------------ + @ThriftClientCall + def getRunData(): + pass + + # ------------------------------------------------------------ + @ThriftClientCall + def getRunResults(self, runId, resultBegin, resultEnd, sortType, reportFilters): + pass + + # ------------------------------------------------------------ + @ThriftClientCall + def getRunResultCount(self, runId, reportFilters): + pass + + #----------------------------------------------------------------------- + @ThriftClientCall + def getRunResultTypes(self, runId, reportFilters): + pass + + #----------------------------------------------------------------------- + @ThriftClientCall + def getAPIVersion(self): + pass + + #----------------------------------------------------------------------- + @ThriftClientCall + def removeRunResults(self, run_ids): + pass + + #----------------------------------------------------------------------- + @ThriftClientCall + def getNewResults(self, base_run_id, new_run_id, limit, offset, sortType, reportFilters): + pass + + #----------------------------------------------------------------------- + @ThriftClientCall + def getUnresolvedResults(self, base_run_id, new_run_id, limit, offset, sortType, reportFilters): + pass + + #----------------------------------------------------------------------- + @ThriftClientCall + def getResolvedResults(self, base_run_id, new_run_id, limit, offset, sortType, reportFilters): + pass Index: tools/codechecker/libcodechecker/context_base.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/context_base.py @@ -0,0 +1,195 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import abc + + +# ----------------------------------------------------------------------------- +class ContextBase(object): + __metaclass__ = abc.ABCMeta + + _package_root = None + _verbose_level = None + _alchemy_log_level = None + _env_path = None + _env_vars = None + _extra_include_paths = [] + _compiler_sysroot = None + _extra_system_include_paths = [] + _codechecker_enable_checkers = set() + _codechecker_disable_checkers = set() + _codechecker_workspace = None + _db_username = None + _module_id = '' + _run_id = None + _severity_map = dict() + + def load_data(self, env_vars, pckg_layout, variables): + self.pckg_layout = pckg_layout + self.variables = variables + self.env_vars = env_vars + + self.set_env(env_vars) + self._db_username = self.variables['default_db_username'] + + @abc.abstractmethod + def set_env(self, env_vars): + self._package_root = os.environ.get(env_vars['env_package_root']) + self._verbose_level = os.environ.get(env_vars['env_verbose_name']) + self._alchemy_log_level = \ + os.environ.get(env_vars['env_alchemy_verbose_name']) + self._env_path = os.environ.get(env_vars['env_path']) + env_enabled_checkers = os.environ.get( + env_vars['codechecker_enable_check']) + if env_enabled_checkers: + self._codechecker_enable_checkers = \ + set(env_enabled_checkers.split(':')) + + env_disabled_checkers = os.environ.get( + env_vars['codechecker_disable_check']) + + if env_disabled_checkers: + self._codechecker_disable_checkers = \ + set(env_disabled_checkers.split(':')) + + codechecker_workspace = os.environ.get('codechecker_workspace') + if codechecker_workspace: + self._codechecker_workspace = codechecker_workspace + + @property + def package_root(self): + return self._package_root + + @property + def verbose_level(self): + return self._verbose_level + + @property + def checker_plugin(self): + return os.path.join(self._package_root, + self.pckg_layout['plugin']) + + @property + def clang_include(self): + return os.path.join(self._package_root, + self.pckg_layout['compiler_include']) + + @property + def extra_includes(self): + return self._extra_include_paths + + @extra_includes.setter + def add_extra_includes(self, path): + self._extra_include_paths.append(path) + + @property + def extra_system_includes(self): + return self._extra_system_include_paths + + @extra_includes.setter + def add_extra_system_includes(self, path): + self._extra_system_include_paths.append(path) + + @property + def gdb_config_file(self): + return os.path.join(self._package_root, + self.pckg_layout['gdb_config_file']) + + @property + def checkers_severity_map_file(self): + return os.path.join(self._package_root, + self.pckg_layout['checkers_severity_map_file']) + + @property + def doc_root(self): + return os.path.join(self._package_root, + self.pckg_layout['docs']) + + @property + def www_root(self): + return os.path.join(self._package_root, + self.pckg_layout['www']) + + @property + def migration_root(self): + return os.path.join(self._package_root, + self.pckg_layout['codechecker_db_migrate']) + + @property + def db_username(self): + return self._db_username + + @db_username.setter + def db_username(self, value): + self._db_username = value + + @property + def pgsql_data_dir_name(self): + return self.variables['pgsql_data_dir_name'] + + @property + def env_enabled_checkers(self): + return self._codechecker_enable_checkers + + @env_enabled_checkers.setter + def env_enabled_checkers(self, value): + self._codechecker_enable_checkers = \ + self._codechecker_enable_checkers.union(value) + + @property + def env_disabled_checkers(self): + return self._codechecker_disable_checkers + + @env_disabled_checkers.setter + def env_disabled_checkers(self, value): + self._codechecker_disable_checkers = \ + self._codechecker_disable_checkers.union(value) + + @property + def codechecker_workspace(self): + return self._codechecker_workspace + + @codechecker_workspace.setter + def codechecker_workspace(self, value): + self._codechecker_workspace = value + + @property + def database_path(self): + return os.path.join(self.codechecker_workspace, + self.pgsql_data_dir_name) + + @property + def compiler_sysroot(self): + return self._compiler_sysroot + + @compiler_sysroot.setter + def compiler_sysroot(self, value): + self._compiler_sysroot = value + + @property + def module_id(self): + return self._module_id + + @module_id.setter + def module_id(self, value): + self._module_id = value + + @property + def run_id(self): + return self._run_id + + @run_id.setter + def run_id(self, value): + self._run_id = value + + @property + def severity_map(self): + return self._severity_map + + @severity_map.setter + def severity_map(self, value): + self._severity_map = value Index: tools/codechecker/libcodechecker/database_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/database_handler.py @@ -0,0 +1,485 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +Database server handling. +''' + +import os +import subprocess +import atexit +import time +import sys + +from abc import ABCMeta, abstractmethod + +from libcodechecker import util +from libcodechecker import logger +from libcodechecker import pgpass +from libcodechecker import host_check + +from libcodechecker.db_model.orm_model import CC_META +from libcodechecker.db_model.orm_model import DBVersion +from libcodechecker.db_model.orm_model import CreateSession + +import sqlalchemy +from sqlalchemy.engine.url import URL, make_url +from sqlalchemy.sql.elements import quoted_name +from sqlalchemy.engine import Engine +from sqlalchemy import event + +from alembic import command, config + +LOG = logger.get_new_logger('DB_HANDLER') + +class SQLServer(object): + ''' + Abstract base class for database server handling. An SQLServer instance is + responsible for the initialization, starting, and stopping the database + server, and also for connection string management. + + SQLServer implementations are created via SQLServer.from_cmdline_args(). + + How to add a new database server implementation: + 1, Derive from SQLServer and implement the abstract methods + 2, Add/modify some command line options in CodeChecker.py + 3, Modify SQLServer.from_cmdline_args() in order to create an + instance of the new server type if needed + ''' + + __metaclass__ = ABCMeta + + + def __init__(self, migration_root): + ''' + Sets self.migration_root. migration_root should be the path to the + alembic migration scripts. + ''' + + self.migration_root = migration_root + + + def _create_or_update_schema(self, use_migration=True): + ''' + Creates or updates the database schema. The database server should be + started before this method is called. + + If use_migration is True, this method runs an alembic upgrade to HEAD. + + In the False case, there is no migration support and only SQLAlchemy + meta data is used for schema creation. + + On error sys.exit(1) is called. + ''' + + try: + db_uri = self.get_connection_string() + engine = SQLServer.create_engine(db_uri) + + LOG.debug('Update/create database schema') + if use_migration: + LOG.debug('Creating new database session') + session = CreateSession(engine) + connection = session.connection() + + cfg = config.Config() + cfg.set_main_option("script_location", self.migration_root) + cfg.attributes["connection"] = connection + command.upgrade(cfg, "head") + + session.commit() + else: + CC_META.create_all(engine) + + engine.dispose() + LOG.debug('Update/create database schema done') + return True + + except sqlalchemy.exc.SQLAlchemyError as alch_err: + LOG.error(str(alch_err)) + sys.exit(1) + + + @abstractmethod + def start(self, db_version_info, wait_for_start=True, init=False): + ''' + Starts the database server and initializes the database server. + + On wait_for_start == True, this method returns when the server is up + and ready for connections. Otherwise it only starts the server and + returns immediately. + + On init == True, this it also initializes the database data and schema + if needed. + + On error sys.exit(1) should be called. + ''' + pass + + + @abstractmethod + def stop(self): + ''' + Terminates the database server. + + On error sys.exit(1) should be called. + ''' + pass + + + @abstractmethod + def get_connection_string(self): + ''' + Returns the connection string for SQLAlchemy. + + DO NOT LOG THE CONNECTION STRING BECAUSE IT MAY CONTAIN THE PASSWORD + FOR THE DATABASE! + ''' + pass + + + @staticmethod + def create_engine(connection_string): + ''' + Creates a new SQLAlchemy engine. + ''' + + if make_url(connection_string).drivername == 'sqlite+pysqlite': + # FIXME: workaround for locking errors + return sqlalchemy.create_engine(connection_string, + encoding='utf8', + connect_args={'timeout': 600}) + else: + return sqlalchemy.create_engine(connection_string, + encoding='utf8') + + @staticmethod + def from_cmdline_args(args, workspace, migration_root, env=None): + ''' + Normally only this method is called form outside of this module in + order to instance the proper server implementation. + + Parameters: + args: the command line arguments from CodeChecker.py + workspace: path to the CodeChecker workspace directory + migration_root: path to the database migration scripts + env: a run environment dictionary. + ''' + + if not host_check.check_sql_driver(args.postgresql): + LOG.error("The selected SQL driver is not available.") + sys.exit(1) + + if args.postgresql: + LOG.debug("Using PostgreSQLServer") + return PostgreSQLServer(workspace, + migration_root, + args.dbaddress, + args.dbport, + args.dbusername, + args.dbname, + run_env=env) + else: + LOG.debug("Using SQLiteDatabase") + return SQLiteDatabase(workspace, migration_root, run_env=env) + + + def check_db_version(self, db_version_info, session=None): + ''' + Checks the database version and prints an error message on database + version mismatch. + + - On mismatching or on missing version a sys.exit(1) is called. + - On missing DBVersion table, it returns False + - On compatible DB version, it returns True + + Parameters: + db_version_info (db_version.DBVersionInfo): required database + version. + session: an open database session or None. If session is None, a + new session is created. + ''' + + try: + dispose_engine = False + if session is None: + engine = SQLServer.create_engine(self.get_connection_string()) + dispose_engine = True + session = CreateSession(engine) + else: + engine = session.get_bind() + + if not engine.has_table(quoted_name(DBVersion.__tablename__, True)): + LOG.debug("Missing DBVersion table!") + return False + + version = session.query(DBVersion).first() + if version is None: + # Version is not populated yet + LOG.error('No version information found in the database.') + sys.exit(1) + elif not db_version_info.is_compatible(version.major, version.minor): + LOG.error('Version mismatch. Expected database version: ' + + str(db_version_info)) + version_from_db = 'v'+str(version.major)+'.'+str(version.minor) + LOG.error('Version from the database is: ' + version_from_db) + LOG.error('Please update your database.') + sys.exit(1) + + LOG.debug("Database version is compatible.") + return True + finally: + session.commit() + if dispose_engine: + engine.dispose() + + + def _add_version(self, db_version_info, session=None): + ''' + Fills the DBVersion table. + ''' + + engine = None + if session is None: + engine = SQLServer.create_engine(self.get_connection_string()) + session = CreateSession(engine) + + expected = db_version_info.get_expected_version() + LOG.debug('Adding DB version: ' + str(expected)) + + session.add(DBVersion(expected[0], expected[1])) + session.commit() + + if engine: + engine.dispose() + + LOG.debug('Adding DB version done!') + +class PostgreSQLServer(SQLServer): + ''' + Handler for PostgreSQL. + ''' + + def __init__(self, workspace, migration_root, host, port, user, database, password = None, run_env=None): + super(PostgreSQLServer, self).__init__(migration_root) + + self.path = os.path.join(workspace, 'pgsql_data') + self.host = host + self.port = port + self.user = user + self.database = database + self.password = password + self.run_env = run_env + self.workspace = workspace + + self.proc = None + + + def _is_database_data_exist(self): + '''Check the PostgreSQL instance existence in a given path.''' + + LOG.debug('Checking for database at ' + self.path) + + return os.path.exists(self.path) and \ + os.path.exists(os.path.join(self.path, 'PG_VERSION')) and \ + os.path.exists(os.path.join(self.path, 'base')) + + + def _initialize_database_data(self): + '''Initialize a PostgreSQL instance with initdb. ''' + + LOG.debug('Initializing database at ' + self.path) + + init_db = ['initdb', '-U', self.user, '-D', self.path, '-E SQL_ASCII'] + + err, code = util.call_command(init_db, self.run_env) + # logger -> print error + return code == 0 + + + def _get_connection_string(self, database): + ''' + Helper method for getting the connection string for the given database. + ''' + + port = str(self.port) + driver = host_check.get_postgresql_driver_name() + password = self.password + if driver == 'pg8000' and not password: + pfilepath = os.environ.get('PGPASSFILE') + if pfilepath: + password = pgpass.get_password_from_file(pfilepath, + self.host, + port, + database, + self.user) + + extra_args = {'client_encoding': 'utf8'} + return str(URL('postgresql+' + driver, + username=self.user, + password=password, + host=self.host, + port=port, + database=database, + query=extra_args)) + + + def _wait_or_die(self): + ''' + Wait for database if the database process was stared + with a different client. No polling is possible. + ''' + + LOG.debug('Waiting for PostgreSQL') + tries_count = 0 + max_try = 20 + timeout = 5 + while not self._is_running() and tries_count < max_try: + tries_count += 1 + time.sleep(timeout) + + if tries_count >= max_try: + LOG.error('Failed to start database.') + sys.exit(1) + + + def _create_database(self): + try: + LOG.debug('Creating new database if not exists') + + db_uri = self._get_connection_string('postgres') + engine = SQLServer.create_engine(db_uri) + text = "SELECT 1 FROM pg_database WHERE datname='%s'" % self.database + if not bool(engine.execute(text).scalar()): + conn = engine.connect() + # From sqlalchemy documentation: + # The psycopg2 and pg8000 dialects also offer the special level AUTOCOMMIT. + conn = conn.execution_options(isolation_level="AUTOCOMMIT") + conn.execute('CREATE DATABASE "%s"' % self.database) + conn.close() + + LOG.debug('Database created: ' + self.database) + + LOG.debug('Database already exists: ' + self.database) + + except sqlalchemy.exc.SQLAlchemyError as alch_err: + LOG.error('Failed to create database!') + LOG.error(str(alch_err)) + sys.exit(1) + + + def _is_running(self): + '''Is there PostgreSQL instance running on a given host and port.''' + + LOG.debug('Checking if database is running at ' + self.host + ':' + str(self.port)) + + check_db = ['psql', '-U', self.user, '-l', '-p', str(self.port), '-h', self.host] + err, code = util.call_command(check_db, self.run_env) + return code == 0 + + + def start(self, db_version_info, wait_for_start=True, init=False): + ''' + Start a PostgreSQL instance with given path, host and port. + Return with process instance + ''' + + LOG.debug('Starting/connecting to database') + if not self._is_running(): + if not util.is_localhost(self.host): + LOG.info('Database is not running yet') + sys.exit(1) + + if not self._is_database_data_exist(): + if not init: + # The database does not exists + LOG.error('Database data is missing!') + LOG.error('Please check your configuration!') + sys.exit(1) + elif not self._initialize_database_data(): + # The database does not exist and cannot create + LOG.error('Database data is missing and the initialization ' + 'of a new failed!') + LOG.error('Please check your configuration!') + sys.exit(1) + + LOG.info('Starting database') + LOG.debug('Starting database at ' + self.host + ':' + str(self.port) + ' ' + self.path) + + db_logfile = os.path.join(self.workspace, 'postgresql.log') \ + if logger.get_log_level() == logger.DEBUG else os.devnull + self._db_log = open(db_logfile, 'wb') + + start_db = ['postgres', '-i', '-D', self.path, '-p', str(self.port), '-h', self.host] + self.proc = subprocess.Popen(start_db, + bufsize=-1, + env=self.run_env, + stdout=self._db_log, + stderr=subprocess.STDOUT) + + add_version = False + if init: + self._wait_or_die() + self._create_database() + add_version = not self.check_db_version(db_version_info) + self._create_or_update_schema() + elif wait_for_start: + self._wait_or_die() + add_version = not self.check_db_version(db_version_info) + + if add_version: + self._add_version(db_version_info) + + atexit.register(self.stop) + LOG.debug('Done') + + + def stop(self): + if self.proc: + LOG.debug('Terminating database') + self.proc.terminate() + self._db_log.close() + + + def get_connection_string(self): + return self._get_connection_string(self.database) + + +class SQLiteDatabase(SQLServer): + ''' + Handler for SQLite. + ''' + def __init__(self, workspace, migration_root, run_env=None): + super(SQLiteDatabase, self).__init__(migration_root) + + self.dbpath = os.path.join(workspace, 'codechecker.sqlite') + self.run_env = run_env + + def _set_sqlite_pragma(dbapi_connection, connection_record): + cursor = dbapi_connection.cursor() + cursor.execute("PRAGMA foreign_keys=ON") + cursor.close() + + event.listen(Engine, 'connect', _set_sqlite_pragma) + + def start(self, db_version_info, wait_for_start=True, init=False): + if init: + add_version = not self.check_db_version(db_version_info) + self._create_or_update_schema(use_migration=False) + if add_version: + self._add_version(db_version_info) + + if not os.path.exists(self.dbpath): + # The database does not exists + LOG.error('Database (%s) is missing!' % self.dbpath) + sys.exit(1) + + + def stop(self): + pass + + + def get_connection_string(self): + return str(URL('sqlite+pysqlite', None, None, None, None, self.dbpath)) Index: tools/codechecker/libcodechecker/db_migrate/env.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/env.py @@ -0,0 +1,80 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +from __future__ import with_statement +from alembic import context +from sqlalchemy import engine_from_config, pool + +# this is the Alembic Config object, which provides +# access to the values within the .ini file in use. +config = context.config + +# add your model's MetaData object here +# for 'autogenerate' support +try: + from libcodechecker.db_model.orm_model import Base +except ImportError: + # Assume we are in the source directory + import sys + import os + sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), + ".."))) + from db_model.orm_model import Base + +target_metadata = Base.metadata + +# other values from the config, defined by the needs of env.py, +# can be acquired: +# my_important_option = config.get_main_option("my_important_option") +# ... etc. + +def run_migrations_offline(): + """Run migrations in 'offline' mode. + + This configures the context with just a URL + and not an Engine, though an Engine is acceptable + here as well. By skipping the Engine creation + we don't even need a DBAPI to be available. + + Calls to context.execute() here emit the given string to the + script output. + + """ + url = config.get_main_option("sqlalchemy.url") + context.configure( + url=url, target_metadata=target_metadata, literal_binds=True) + + with context.begin_transaction(): + context.run_migrations() + + +def run_migrations_online(): + """Run migrations in 'online' mode. + + In this scenario we need to create an Engine + and associate a connection with the context. + + """ + connectable = config.attributes.get('connection', None) + if connectable is None: + connectable = engine_from_config( + config.get_section(config.config_ini_section), + prefix='sqlalchemy.', + poolclass=pool.NullPool) + + with connectable.connect() as connection: + context.configure( + connection=connection, + target_metadata=target_metadata + ) + + with context.begin_transaction(): + context.run_migrations() + +if context.is_offline_mode(): + run_migrations_offline() +else: + run_migrations_online() Index: tools/codechecker/libcodechecker/db_migrate/script.py.mako =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/script.py.mako @@ -0,0 +1,24 @@ +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision | comma,n} +Create Date: ${create_date} + +""" + +# revision identifiers, used by Alembic. +revision = ${repr(up_revision)} +down_revision = ${repr(down_revision)} +branch_labels = ${repr(branch_labels)} +depends_on = ${repr(depends_on)} + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +def upgrade(): + ${upgrades if upgrades else "pass"} + + +def downgrade(): + ${downgrades if downgrades else "pass"} Index: tools/codechecker/libcodechecker/db_migrate/versions/2b23d1a4fb96_udpate_suppress_handling.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/versions/2b23d1a4fb96_udpate_suppress_handling.py @@ -0,0 +1,76 @@ +"""udpate suppress handling + +Revision ID: 2b23d1a4fb96 +Revises: 63efc03c2a5 +Create Date: 2015-10-06 12:40:06.669407 + +""" + +# revision identifiers, used by Alembic. +revision = '2b23d1a4fb96' +down_revision = '63efc03c2a5' +branch_labels = None +depends_on = None + +import sys + +from alembic import op +import sqlalchemy as sa +from sqlalchemy.schema import Sequence, CreateSequence + +# NOTICE! +# kept here only for later reference if required for other +# db upgrades if full table copy or data modification is required +# +# helper table +#suppress_bug_helper = sa.Table( +# 'suppress_bug', +# sa.MetaData(), +# sa.Column('file_name', sa.String()), +# sa.Column('hash', sa.String()), +#) +# NOTICE END + +def upgrade(): + + op.drop_constraint('pk_suppress_bug', 'suppress_bug', type_='primary') + op.add_column('suppress_bug', + sa.Column('file_name', sa.String(), nullable=True)) + # add new id primary key + op.add_column('suppress_bug', + sa.Column('id', sa.Integer(), autoincrement=True, primary_key=True)) + + # create constraint + op.create_primary_key( + "pk_suppress_bug", "suppress_bug", + ["id"] + ) + + # required to fill up autoincrement values for the id + op.execute(CreateSequence(Sequence('suppress_bug_id'))) + op.alter_column("suppress_bug", + "id", + nullable=False, + server_default=sa.text("nextval('suppress_bug_id'::regclass)")) + + + # NOTICE! + # kept here only for later reference if required for other + # db upgrades if full table copy or data modification is required + # copy the full table and during the copy modify values if required + # use the current connection + #connection = op.get_bind() + + #for sp_bug in connection.execute(suppress_bug_helper.select()): + # connection.execute( + # suppress_bug_helper.update().where( + # suppress_bug_helper.c.hash == sp_bug.hash + # ).values( + # file_name='', + # ) + # ) + # NOTICE END + +def downgrade(): + # downgrade is not supported + sys.exit(1) Index: tools/codechecker/libcodechecker/db_migrate/versions/30e41fdf2e85_store_analyzer_type_and_analyzed_source_.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/versions/30e41fdf2e85_store_analyzer_type_and_analyzed_source_.py @@ -0,0 +1,48 @@ +"""Store analyzer type and analyzed source file to the database for each buildaction + +Revision ID: 30e41fdf2e85 +Revises: 4e97419519b3 +Create Date: 2016-07-04 15:36:26.208047 + +""" + +# revision identifiers, used by Alembic. +revision = '30e41fdf2e85' +down_revision = '4e97419519b3' +branch_labels = None +depends_on = None + +import sys + +from alembic import op +import sqlalchemy as sa + + +def upgrade(): + ''' + extend build_actions table with columns to identify if + the results for a build_action should be deleted in update mode + + analyzer_type: is required to identify the analyzer which analyzer analyzed the build action + + analyzed_source_file: is required to identify which source file was analyzed in the build action (it is possible to contain multiple source files) + ''' + + op.add_column('build_actions', + sa.Column('analyzed_source_file', + sa.String(), + nullable=False, + server_default='') + ) + + op.add_column('build_actions', + sa.Column('analyzer_type', + sa.String(), + nullable=False, + server_default='') + ) + + +def downgrade(): + # downgrade is not supported + sys.exit(1) Index: tools/codechecker/libcodechecker/db_migrate/versions/453083da7cce_quick_delete.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/versions/453083da7cce_quick_delete.py @@ -0,0 +1,24 @@ +"""quick delete + +Revision ID: 453083da7cce +Revises: f6791c2b40d +Create Date: 2015-11-26 15:11:36.790627 + +""" + +# revision identifiers, used by Alembic. +revision = '453083da7cce' +down_revision = 'f6791c2b40d' +branch_labels = None +depends_on = None + +from alembic import op +import sys + +def upgrade(): + op.create_index(op.f('ix_reports_end_bugevent'), 'reports', ['end_bugevent'], unique=False) + op.create_index(op.f('ix_reports_start_bugevent'), 'reports', ['start_bugevent'], unique=False) + + +def downgrade(): + sys.exit(-1) Index: tools/codechecker/libcodechecker/db_migrate/versions/464168cc48ad_remove_bug_hash_type.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/versions/464168cc48ad_remove_bug_hash_type.py @@ -0,0 +1,32 @@ +"""remove_bug_hash_type + +Revision ID: 464168cc48ad +Revises: 2b23d1a4fb96 +Create Date: 2015-11-04 17:45:32.136366 + +""" + +# revision identifiers, used by Alembic. +revision = '464168cc48ad' +down_revision = '2b23d1a4fb96' +branch_labels = None +depends_on = None + +import sys + +from alembic import op + + +def upgrade(): + """ + remove bug hash type + """ + op.drop_column('reports', 'bug_id_type') + op.drop_column('suppress_bug', 'type') + + +def downgrade(): + """ + downgrade is not supported + """ + sys.exit(1) Index: tools/codechecker/libcodechecker/db_migrate/versions/4e97419519b3_remove_run_id_primary_key_from_skippath_.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/versions/4e97419519b3_remove_run_id_primary_key_from_skippath_.py @@ -0,0 +1,37 @@ +"""Remove run_id primary key from SkipPath table + +Revision ID: 4e97419519b3 +Revises: 453083da7cce +Create Date: 2016-02-12 13:34:23.302931 + +""" + +# revision identifiers, used by Alembic. +revision = '4e97419519b3' +down_revision = '453083da7cce' +branch_labels = None +depends_on = None + +import sys +from alembic import op +import sqlalchemy as sa + + +def upgrade(): + """ + remove primary key constraint for run_id in the skip path table + """ + op.drop_constraint('pk_skip_path', 'skip_path', type_='primary') + + # create new primary key constraint for the id only + op.create_primary_key( + "pk_skip_path", "skip_path", + ["id"] + ) + + op.alter_column('skip_path', + sa.Column('run_id', sa.Integer(), nullable=False)) + +def downgrade(): + # downgrade is not supported + sys.exit(1) Index: tools/codechecker/libcodechecker/db_migrate/versions/63efc03c2a5_initial_schema.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/versions/63efc03c2a5_initial_schema.py @@ -0,0 +1,154 @@ +"""Initial schema + +Revision ID: 63efc03c2a5 +Revises: +Create Date: 2015-09-25 14:31:57.381866 + +""" + +# revision identifiers, used by Alembic. +revision = '63efc03c2a5' +down_revision = None +branch_labels = None +depends_on = None + +from alembic import op +import sqlalchemy as sa + + +def upgrade(): + op.create_table('db_version', + sa.Column('major', sa.Integer(), nullable=False), + sa.Column('minor', sa.Integer(), nullable=False), + sa.PrimaryKeyConstraint('major', 'minor', name=op.f('pk_db_version')) + ) + op.create_table('runs', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('date', sa.DateTime(), nullable=True), + sa.Column('duration', sa.Integer(), nullable=True), + sa.Column('name', sa.String(), nullable=True), + sa.Column('version', sa.String(), nullable=True), + sa.Column('command', sa.String(), nullable=True), + sa.Column('inc_count', sa.Integer(), nullable=True), + sa.PrimaryKeyConstraint('id', name=op.f('pk_runs')), + sa.UniqueConstraint('name', name=op.f('uq_runs_name')) + ) + op.create_table('build_actions', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('run_id', sa.Integer(), nullable=True), + sa.Column('build_cmd', sa.String(), nullable=True), + sa.Column('check_cmd', sa.String(), nullable=True), + sa.Column('failure_txt', sa.String(), nullable=True), + sa.Column('date', sa.DateTime(), nullable=True), + sa.Column('duration', sa.Integer(), nullable=True), + sa.ForeignKeyConstraint(['run_id'], [u'runs.id'], name=op.f('fk_build_actions_run_id_runs'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('id', name=op.f('pk_build_actions')) + ) + op.create_table('configs', + sa.Column('run_id', sa.Integer(), nullable=False), + sa.Column('checker_name', sa.String(), nullable=False), + sa.Column('attribute', sa.String(), nullable=False), + sa.Column('value', sa.String(), nullable=False), + sa.ForeignKeyConstraint(['run_id'], [u'runs.id'], name=op.f('fk_configs_run_id_runs'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('run_id', 'checker_name', 'attribute', 'value', name=op.f('pk_configs')) + ) + op.create_table('files', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('run_id', sa.Integer(), nullable=True), + sa.Column('filepath', sa.String(), nullable=True), + sa.Column('content', sa.Binary(), nullable=True), + sa.Column('inc_count', sa.Integer(), nullable=True), + sa.ForeignKeyConstraint(['run_id'], [u'runs.id'], name=op.f('fk_files_run_id_runs'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('id', name=op.f('pk_files')) + ) + op.create_table('skip_path', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('path', sa.String(), nullable=True), + sa.Column('run_id', sa.Integer(), nullable=False), + sa.Column('comment', sa.Binary(), nullable=True), + sa.ForeignKeyConstraint(['run_id'], [u'runs.id'], name=op.f('fk_skip_path_run_id_runs'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('id', 'run_id', name=op.f('pk_skip_path')) + ) + op.create_table('suppress_bug', + sa.Column('hash', sa.String(), nullable=False), + sa.Column('run_id', sa.Integer(), nullable=False), + sa.Column('type', sa.Integer(), nullable=True), + sa.Column('comment', sa.Binary(), nullable=True), + sa.ForeignKeyConstraint(['run_id'], [u'runs.id'], name=op.f('fk_suppress_bug_run_id_runs'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('hash', 'run_id', name=op.f('pk_suppress_bug')) + ) + op.create_table('bug_path_events', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('line_begin', sa.Integer(), nullable=True), + sa.Column('col_begin', sa.Integer(), nullable=True), + sa.Column('line_end', sa.Integer(), nullable=True), + sa.Column('col_end', sa.Integer(), nullable=True), + sa.Column('msg', sa.String(), nullable=True), + sa.Column('file_id', sa.Integer(), nullable=True), + sa.Column('next', sa.Integer(), nullable=True), + sa.Column('prev', sa.Integer(), nullable=True), + sa.ForeignKeyConstraint(['file_id'], [u'files.id'], name=op.f('fk_bug_path_events_file_id_files'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('id', name=op.f('pk_bug_path_events')) + ) + op.create_index(op.f('ix_bug_path_events_file_id'), 'bug_path_events', ['file_id'], unique=False) + op.create_table('bug_report_points', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('line_begin', sa.Integer(), nullable=True), + sa.Column('col_begin', sa.Integer(), nullable=True), + sa.Column('line_end', sa.Integer(), nullable=True), + sa.Column('col_end', sa.Integer(), nullable=True), + sa.Column('file_id', sa.Integer(), nullable=True), + sa.Column('next', sa.Integer(), nullable=True), + sa.ForeignKeyConstraint(['file_id'], [u'files.id'], name=op.f('fk_bug_report_points_file_id_files'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('id', name=op.f('pk_bug_report_points')) + ) + op.create_index(op.f('ix_bug_report_points_file_id'), 'bug_report_points', ['file_id'], unique=False) + op.create_table('reports', + sa.Column('id', sa.Integer(), nullable=False), + sa.Column('file_id', sa.Integer(), nullable=True), + sa.Column('run_id', sa.Integer(), nullable=True), + sa.Column('bug_id', sa.String(), nullable=True), + sa.Column('bug_id_type', sa.Integer(), nullable=True), + sa.Column('checker_id', sa.String(), nullable=True), + sa.Column('checker_cat', sa.String(), nullable=True), + sa.Column('bug_type', sa.String(), nullable=True), + sa.Column('severity', sa.Integer(), nullable=True), + sa.Column('checker_message', sa.String(), nullable=True), + sa.Column('start_bugpoint', sa.Integer(), nullable=True), + sa.Column('start_bugevent', sa.Integer(), nullable=True), + sa.Column('end_bugevent', sa.Integer(), nullable=True), + sa.Column('suppressed', sa.Boolean(), nullable=True), + sa.ForeignKeyConstraint(['end_bugevent'], [u'bug_path_events.id'], name=op.f('fk_reports_end_bugevent_bug_path_events'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.ForeignKeyConstraint(['file_id'], [u'files.id'], name=op.f('fk_reports_file_id_files'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.ForeignKeyConstraint(['run_id'], [u'runs.id'], name=op.f('fk_reports_run_id_runs'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.ForeignKeyConstraint(['start_bugevent'], [u'bug_path_events.id'], name=op.f('fk_reports_start_bugevent_bug_path_events'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.ForeignKeyConstraint(['start_bugpoint'], [u'bug_report_points.id'], name=op.f('fk_reports_start_bugpoint_bug_report_points'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('id', name=op.f('pk_reports')) + ) + op.create_index(op.f('ix_reports_bug_id'), 'reports', ['bug_id'], unique=False) + op.create_index(op.f('ix_reports_run_id'), 'reports', ['run_id'], unique=False) + op.create_table('reports_to_build_actions', + sa.Column('report_id', sa.Integer(), nullable=False), + sa.Column('build_action_id', sa.Integer(), nullable=False), + sa.ForeignKeyConstraint(['build_action_id'], [u'build_actions.id'], name=op.f('fk_reports_to_build_actions_build_action_id_build_actions'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.ForeignKeyConstraint(['report_id'], [u'reports.id'], name=op.f('fk_reports_to_build_actions_report_id_reports'), ondelete=u'CASCADE', initially=u'DEFERRED', deferrable=True), + sa.PrimaryKeyConstraint('report_id', 'build_action_id', name=op.f('pk_reports_to_build_actions')) + ) + + +def downgrade(): + op.drop_table('reports_to_build_actions') + op.drop_index(op.f('ix_reports_run_id'), table_name='reports') + op.drop_index(op.f('ix_reports_bug_id'), table_name='reports') + op.drop_table('reports') + op.drop_index(op.f('ix_bug_report_points_file_id'), table_name='bug_report_points') + op.drop_table('bug_report_points') + op.drop_index(op.f('ix_bug_path_events_file_id'), table_name='bug_path_events') + op.drop_table('bug_path_events') + op.drop_table('suppress_bug') + op.drop_table('skip_path') + op.drop_table('files') + op.drop_table('configs') + op.drop_table('build_actions') + op.drop_table('runs') + op.drop_table('db_version') Index: tools/codechecker/libcodechecker/db_migrate/versions/f6791c2b40d_add_can_delete_to_run_table.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_migrate/versions/f6791c2b40d_add_can_delete_to_run_table.py @@ -0,0 +1,27 @@ +"""Add can_delete to Run table + +Revision ID: f6791c2b40d +Revises: 464168cc48ad +Create Date: 2015-11-17 16:22:02.793689 + +""" + +# revision identifiers, used by Alembic. +revision = 'f6791c2b40d' +down_revision = '464168cc48ad' +branch_labels = None +depends_on = None + +import sys + +from alembic import op +import sqlalchemy as sa + + +def upgrade(): + op.add_column('runs', + sa.Column('can_delete', sa.Boolean(), nullable=False, server_default=sa.sql.expression.true(), default=True)) + +def downgrade(): + # downgrade is not supported + sys.exit(1) Index: tools/codechecker/libcodechecker/db_model/__init__.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_model/__init__.py @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. Index: tools/codechecker/libcodechecker/db_model/orm_model.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_model/orm_model.py @@ -0,0 +1,284 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +ORM model. +''' +from __future__ import print_function +from __future__ import unicode_literals + +from sqlalchemy import * +from sqlalchemy.orm import * +from sqlalchemy.sql.expression import true +from sqlalchemy.ext.declarative import declarative_base + +from datetime import datetime +from math import ceil + +CC_META = MetaData(naming_convention={ + "ix": 'ix_%(column_0_label)s', + "uq": "uq_%(table_name)s_%(column_0_name)s", + "ck": "ck_%(table_name)s_%(column_0_name)s", + "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s", + "pk": "pk_%(table_name)s" +}) + +# Create base class for ORM classes +Base = declarative_base(metadata=CC_META) + +# Start of ORM classes + +class DBVersion(Base): + __tablename__ = 'db_version' + # TODO: constraint, only one line in this table + major = Column(Integer, primary_key=True) + minor = Column(Integer, primary_key=True) + + def __init__(self, major, minor): + self.major = major + self.minor = minor + + +class Run(Base): + __tablename__ = 'runs' + + __table_args__ = ( + UniqueConstraint('name'), + ) + + id = Column(Integer, autoincrement=True, primary_key=True) + date = Column(DateTime) + duration = Column(Integer) # Seconds, -1 if unfinished. + name = Column(String) + version = Column(String) + command = Column(String) + inc_count = Column(Integer) + can_delete = Column(Boolean, nullable=False, server_default=true(), default=True) + + # Relationships (One to Many). + configlist = relationship('Config', cascade="all, delete-orphan", passive_deletes=True) + buildactionlist = relationship('BuildAction', cascade="all, delete-orphan", passive_deletes=True) + + def __init__(self, name, version, command): + self.date, self.name, self.version, self.command = \ + datetime.now(), name, version, command + self.duration = -1 + self.inc_count = 0 + + def mark_finished(self): + self.duration = ceil((datetime.now() - self.date).total_seconds()) + + +class Config(Base): + __tablename__ = 'configs' + + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), primary_key=True) + checker_name = Column(String, primary_key=True) + attribute = Column(String, primary_key=True) + value = Column(String, primary_key=True) + + def __init__(self, run_id, checker_name, attribute, value): + self.attribute, self.value = attribute, value + self.checker_name, self.run_id = checker_name, run_id + +class File(Base): + __tablename__ = 'files' + + id = Column(Integer, autoincrement=True, primary_key=True) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + filepath = Column(String) + content = Column(Binary) + inc_count = Column(Integer) + + def __init__(self, run_id, filepath): + self.run_id, self.filepath = run_id, filepath + self.inc_count = 0 + + def addContent(self, content): + self.content = content + + +class BuildAction(Base): + __tablename__ = 'build_actions' + + id = Column(Integer, autoincrement=True, primary_key=True) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + build_cmd = Column(String) + analyzer_type = Column(String, nullable=False) + analyzed_source_file = Column(String, nullable=False) + check_cmd = Column(String) + # No failure if the text is empty. + failure_txt = Column(String) + date = Column(DateTime) + + # Seconds, -1 if unfinished. + duration = Column(Integer) + + def __init__(self, run_id, build_cmd, check_cmd, analyzer_type, analyzed_source_file): + self.run_id, self.build_cmd, self.check_cmd, self.failure_txt = \ + run_id, build_cmd, check_cmd, '' + self.date = datetime.now() + self.analyzer_type = analyzer_type + self.analyzed_source_file = analyzed_source_file + self.duration = -1 + + def mark_finished(self, failure_txt): + self.failure_txt = failure_txt + self.duration = (datetime.now() - self.date).total_seconds() + + +class BugPathEvent(Base): + __tablename__ = 'bug_path_events' + + id = Column(Integer, autoincrement=True, primary_key=True) + line_begin = Column(Integer) + col_begin = Column(Integer) + line_end = Column(Integer) + col_end = Column(Integer) + msg = Column(String) + file_id = Column(Integer, ForeignKey('files.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + + next = Column(Integer) + prev = Column(Integer) + + def __init__(self, line_begin, col_begin, line_end, col_end, msg, file_id): + self.line_begin, self.col_begin, self.line_end, self.col_end, self.msg = \ + line_begin, col_begin, line_end, col_end, msg + self.file_id = file_id + + def addPrev(self, prev): + self.prev = prev + + def addNext(self, next): + self.next = next + + def isLast(self): + return self.next is None + + def isFirst(self): + return self.prev is None + + + +class BugReportPoint(Base): + __tablename__ = 'bug_report_points' + + id = Column(Integer, autoincrement=True, primary_key=True) + line_begin = Column(Integer) + col_begin = Column(Integer) + line_end = Column(Integer) + col_end = Column(Integer) + file_id = Column(Integer, ForeignKey('files.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + + # TODO: Add check, the value must be an existing id or null. + # Be careful when inserting. + next = Column(Integer) + + def __init__(self, line_begin, col_begin, line_end, col_end, file_id): + self.line_begin, self.col_begin, self.line_end, self.col_end = \ + line_begin, col_begin, line_end, col_end + self.file_id = file_id + + def addNext(self, next): + self.next = next + + def isLast(self): + return self.next is None + + +class Report(Base): + __tablename__ = 'reports' + + id = Column(Integer, autoincrement=True, primary_key=True) + file_id = Column(Integer, ForeignKey('files.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + bug_id = Column(String, index = True) + checker_id = Column(String) + checker_cat = Column(String) + bug_type = Column(String) + severity = Column(Integer) + + # TODO: multiple messages to multiple source locations? + checker_message = Column(String) + start_bugpoint = Column(Integer, ForeignKey('bug_report_points.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + + start_bugevent = Column(Integer, ForeignKey('bug_path_events.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + end_bugevent = Column(Integer, ForeignKey('bug_path_events.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + suppressed = Column(Boolean) + + # Cascade delete might remove rows SQLAlchemy warns about this + # to remove warnings about already deleted items set this to False. + __mapper_args__ = { + 'confirm_deleted_rows' : False + } + + + # Priority/severity etc... + def __init__(self, run_id, bug_id, file_id, checker_message, start_bugpoint, start_bugevent, end_bugevent, checker_id, checker_cat, bug_type, severity, suppressed): + self.run_id = run_id + self.file_id = file_id + self.bug_id, self.checker_message = bug_id, checker_message + self.start_bugpoint = start_bugpoint + self.start_bugevent = start_bugevent + self.end_bugevent = end_bugevent + self.severity = severity + self.checker_id, self.checker_cat, self.bug_type = checker_id, checker_cat, bug_type + self.suppressed = suppressed + +class ReportsToBuildActions(Base): + __tablename__ = 'reports_to_build_actions' + + report_id = Column(Integer, ForeignKey('reports.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), primary_key=True) + build_action_id = Column( + Integer, ForeignKey('build_actions.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), primary_key=True) + + def __init__(self, report_id, build_action_id): + self.report_id = report_id + self.build_action_id = build_action_id + + +class SuppressBug(Base): + __tablename__ = 'suppress_bug' + + id = Column(Integer, autoincrement=True, primary_key=True) + hash = Column(String, nullable=False) + file_name = Column(String) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), nullable=False) + comment = Column(Binary) + + def __init__(self, run_id, hash, file_name, comment): + self.hash, self.run_id = hash, run_id + self.comment = comment + self.file_name = file_name + + +class SkipPath(Base): + __tablename__ = 'skip_path' + + id = Column(Integer, autoincrement=True, primary_key=True) + path = Column(String) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), nullable=False) + comment = Column(Binary) + + def __init__(self, run_id, path, comment): + self.path = path + self.run_id = run_id + self.comment = comment + +# End of ORM classes. + + +def CreateSchema(engine): + """ Creates the schema if it does not exists. + Do not check version or do migration yet. """ + Base.metadata.create_all(engine) + + +def CreateSession(engine): + """ Creates a scoped session factory that can act like a session. + The factory uses a thread_local registry, so every thread have + its own session. """ + SessionFactory = scoped_session(sessionmaker(bind=engine)) + return SessionFactory Index: tools/codechecker/libcodechecker/db_version.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/db_version.py @@ -0,0 +1,21 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + + +class DBVersionInfo(object): + + def __init__(self, major, minor): + self.major = int(major) + self.minor = int(minor) + + def is_compatible(self, major, minor): + return major == self.major + + def get_expected_version(self): + return (self.major, self.minor) + + def __str__(self): + return 'v'+str(self.major)+'.'+str(self.minor) Index: tools/codechecker/libcodechecker/debug_reporter.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/debug_reporter.py @@ -0,0 +1,93 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import sys + +import sqlalchemy +from sqlalchemy.sql import and_ + +from db_model.orm_model import BuildAction +from db_model.orm_model import Run + +from libcodechecker import analyzer_env +from libcodechecker import logger +from libcodechecker import analyzer_crash_handler +from libcodechecker import database_handler + +LOG = logger.get_new_logger('DEBUG_REPORTER') + + +# ----------------------------------------------------------------------------- +def get_dump_file_name(run_id, action_id): + return 'action_' + str(run_id) + '_' + str(action_id) + '_dump.log' + + +# ----------------------------------------------------------------------------- +def debug(context, connection_string, force): + try: + engine = database_handler.SQLServer.create_engine(connection_string) + session = sqlalchemy.orm.scoped_session( + sqlalchemy.orm.sessionmaker(bind=engine)) + + # Get latest run id + last_run = session.query(Run).order_by(Run.id.desc()).first() + + # Get all failed actions + actions = session.query(BuildAction).filter(and_( + BuildAction.run_id == last_run.id, + sqlalchemy.sql.func.length(BuildAction.failure_txt) != 0)) + + debug_env = analyzer_env.get_check_env(context.path_env_extra, + context.ld_lib_path_extra) + + crash_handler = analyzer_crash_handler.AnalyzerCrashHandler(context, + debug_env) + + dumps_dir = context.dump_output_dir + if not os.path.exists(dumps_dir): + os.mkdir(dumps_dir) + + LOG.info('Generating gdb dump files to : ' + dumps_dir) + + for action in actions: + LOG.info('Processing action ' + str(action.id) + '.') + debug_log_file = \ + os.path.join(dumps_dir, get_dump_file_name(last_run.id, action.id)) + if not force and os.path.exists(debug_log_file): + LOG.info('This file already exists.') + continue + + LOG.info('Generating stacktrace with gdb.') + + gdb_result = \ + crash_handler.get_crash_info(str(action.check_cmd).split()) + + LOG.info('Writing debug info to file.') + + with open(debug_log_file, 'w') as log_file: + log_file.write('========================\n') + log_file.write('Original build command: \n') + log_file.write('========================\n') + log_file.write(action.build_cmd + '\n') + log_file.write('===============\n') + log_file.write('Check command: \n') + log_file.write('===============\n') + log_file.write(action.check_cmd + '\n') + log_file.write('==============\n') + log_file.write('Failure text: \n') + log_file.write('==============\n') + log_file.write(action.failure_txt + '\n') + log_file.write('==========\n') + log_file.write('GDB info: \n') + log_file.write('==========\n') + log_file.write(gdb_result) + + LOG.info('All new debug files are placed in ' + dumps_dir) + + except KeyboardInterrupt as kb_exc: + LOG.error(str(kb_exc)) + sys.exit(1) Index: tools/codechecker/libcodechecker/decorators.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/decorators.py @@ -0,0 +1,46 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import time + +import shared +import sqlalchemy + + +def timeit(method): + def timed(*args, **kw): + timer_begin = time.time() + result = method(*args, **kw) + timer_end = time.time() + + print '%r (%r, %r) %2.2f sec' % (method.__name__, args, kw, + timer_end - timer_begin) + return result + + return timed + + +def trace(method): + def wrapped(*args, **kw): + print('Stepped into ' + method.__name__) + result = method(*args, **kw) + + print('Stepped out ' + method.__name__) + return result + + return wrapped + + +def catch_sqlalchemy(method): + def wrapped(*args, **kw): + try: + return method(*args, **kw) + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, + msg) + + return wrapped Index: tools/codechecker/libcodechecker/generic_package_context.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/generic_package_context.py @@ -0,0 +1,247 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import sys +import json + +from libcodechecker import context_base +from libcodechecker import logger +from libcodechecker import db_version + +from libcodechecker.analyzers import analyzer_types + +LOG = logger.get_new_logger('CONTEXT') + + +# ----------------------------------------------------------------------------- +class Context(context_base.ContextBase): + ''' generic package specific context''' + + __instance = None + + logger_bin = None + logger_file = None + logger_compilers = None + ld_preload = None + __package_version = None + __package_root = None + + def __init__(self, package_root, pckg_layout, cfg_dict): + + env_vars = cfg_dict['environment_variables'] + variables = cfg_dict['package_variables'] + self.__checker_config = cfg_dict['checker_config'] + + # Base class constructor gets the common environment variables + super(Context, self).__init__() + super(Context, self).load_data(env_vars, pckg_layout, variables) + + # get package specific environment variables + self.set_env(env_vars) + + self.__package_root = package_root + + version, database_version = self.__get_version(self.version_file) + self.__package_version = version + self.__db_version_info = db_version.DBVersionInfo(database_version['major'], + database_version['minor']) + Context.__instance = self + + def set_env(self, env_vars): + ''' + get the environment variables + ''' + super(Context, self).set_env(env_vars) + + # get generic package specific environment variables + self.logger_bin = os.environ.get(env_vars['cc_logger_bin']) + self.logger_file = os.environ.get(env_vars['cc_logger_file']) + self.logger_compilers = os.environ.get(env_vars['cc_logger_compiles']) + self.ld_preload = os.environ.get(env_vars['ld_preload']) + self.ld_lib_path = env_vars['env_ld_lib_path'] + + def __get_version(self, version_file): + ''' + get the package version fron the verison config file + ''' + try: + with open(version_file, 'r') as vfile: + vfile_data = json.loads(vfile.read()) + + package_version = vfile_data['version'] + db_version = vfile_data['db_version'] + + except ValueError as verr: + # db_version is required to know if the db schema is compatible + LOG.error('Failed to get version info from the version file') + LOG.error(verr) + sys.exit(1) + + return package_version, db_version + + @property + def default_checkers_config(self): + return self.__checker_config + + @property + def version(self): + return self.__package_version + + @property + def db_version_info(self): + return self.__db_version_info + + @property + def version_file(self): + return os.path.join(self.__package_root, self.pckg_layout['version_file']) + + @property + def env_var_cc_logger_bin(self): + return self.env_vars['cc_logger_bin'] + + @property + def env_var_ld_preload(self): + return self.env_vars['ld_preload'] + + @property + def env_var_cc_logger_compiles(self): + return self.env_vars['cc_logger_compiles'] + + @property + def env_var_cc_logger_file(self): + return self.env_vars['cc_logger_file'] + + @property + def path_logger_bin(self): + return os.path.join(self.package_root, + self.pckg_layout['ld_logger_bin']) + + @property + def path_logger_lib(self): + return os.path.join(self.package_root, + self.pckg_layout['ld_logger_lib_path']) + + @property + def logger_lib_name(self): + return self.pckg_layout['ld_logger_lib_name'] + + @property + def dumps_dir_name(self): + return self.variables['path_dumps_name'] + + @property + def pg_data_dir(self): + return os.path.join(self.codechecker_workspace, + self.pgsql_data_dir_name) + + @property + def dump_output_dir(self): + return os.path.join(self.codechecker_workspace, + self.variables['path_dumps_name']) + + @property + def compiler_resource_dirs(self): + compiler_resource_dirs = self.pckg_layout.get('compiler_resource_dirs') + if not compiler_resource_dirs: + return [] + else: + inc_dirs = [] + for path in compiler_resource_dirs: + if os.path.isabs(path): + inc_dirs.append(path) + else: + inc_dirs.append(os.path.join(self.__package_root, path)) + return inc_dirs + + @property + def path_env_extra(self): + extra_paths = self.pckg_layout.get('path_env_extra') + if not extra_paths: + return [] + else: + paths = [] + for path in extra_paths: + if os.path.isabs(path): + paths.append(path) + else: + paths.append(os.path.join(self.__package_root, path)) + return paths + + @property + def ld_lib_path_extra(self): + extra_lib = self.pckg_layout.get('ld_lib_path_extra') + if not extra_lib: + return [] + else: + ld_paths = [] + for path in extra_lib: + if os.path.isabs(path): + ld_paths.append(path) + else: + ld_paths.append(os.path.join(self.__package_root, path)) + return ld_paths + + + @property + def analyzer_binaries(self): + analyzers = {} + + compiler_binaries = self.pckg_layout.get('analyzers') + if not compiler_binaries: + # set default analyzers assume they are in the PATH + # will be checked later + # key naming in the dict should be the same as in + # the supported analyzers list + analyzers[analyzer_types.CLANG_SA] = 'clang' + analyzers[analyzer_types.CLANG_TIDY] = 'clang-tidy' + else: + for name, value in compiler_binaries.iteritems(): + if os.path.isabs(value): + # check if it is an absolute path + analyzers[name] = value + elif os.path.dirname(value): + # check if it is a package relative path + analyzers[name] = os.path.join(self.__package_root, value) + else: + analyzers[name] = value + + return analyzers + +def get_context(): + + LOG.debug('Loading package config') + + package_root = os.environ['CC_PACKAGE_ROOT'] + + pckg_config_file = os.path.join(package_root, "config", "config.json") + LOG.debug('Reading config: ' + pckg_config_file) + with open(pckg_config_file, 'r') as cfg: + cfg_dict = json.loads(cfg.read()) + + LOG.debug(cfg_dict) + + LOG.debug('Loading layout config') + + layout_cfg_file = os.path.join(package_root, "config", "package_layout.json") + LOG.debug(layout_cfg_file) + with open(layout_cfg_file, 'r') as lcfg: + lcfg_dict = json.loads(lcfg.read()) + + # merge static and runtime layout + layout_config = lcfg_dict['static'].copy() + layout_config.update(lcfg_dict['runtime']) + + LOG.debug(layout_config) + + try: + return Context(package_root, + layout_config, + cfg_dict) + + except KeyError as kerr: + LOG.error(kerr) + sys.exit(1) Index: tools/codechecker/libcodechecker/generic_package_suppress_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/generic_package_suppress_handler.py @@ -0,0 +1,39 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +handler for suppressing a bug +''' + +from libcodechecker import logger +from libcodechecker import suppress_handler +from libcodechecker import suppress_file_handler + +# Warning! this logger should only be used in this module +LOG = logger.get_new_logger('SUPPRESS') + + +class GenericSuppressHandler(suppress_handler.SuppressHandler): + + def store_suppress_bug_id(self, bug_id, file_name, comment): + + if self.suppress_file is None: + return False + + ret = suppress_file_handler.write_to_suppress_file(self.suppress_file, + bug_id, + file_name, + comment) + return ret + + def remove_suppress_bug_id(self, bug_id, file_name): + + if self.suppress_file is None: + return False + + ret = suppress_file_handler.remove_from_suppress_file(self.suppress_file, + bug_id, + file_name) + return ret Index: tools/codechecker/libcodechecker/host_check.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/host_check.py @@ -0,0 +1,136 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import errno +import subprocess +import os + +from libcodechecker import logger + +LOG = logger.get_new_logger('HOST CHECK') + + +# ----------------------------------------------------------------------------- +def check_zlib(): + ''' Check if zlib compression is available + if wrong libraries are installed on the host machine it is + possible the the compression failes which is required to + store data into the database. + ''' + + try: + import zlib + return True + except Exception as ex: + LOG.error(str(ex)) + LOG.error('Failed to import zlib module') + return False + + try: + zlib.compress('Compress this') + return True + except Exception as ex: + LOG.error(str(ex)) + LOG.error('Zlib copression error', zlib.Z_BEST_COMPRESSION) + return False + +# ----------------------------------------------------------------------------- +def get_postgresql_driver_name(): + try: + driver = os.getenv('CODECHECKER_DB_DRIVER') + if driver: + return driver + + try: + import psycopg2 # NOQA + return "psycopg2" + except Exception: + import pg8000 # NOQA + return "pg8000" + except Exception as ex: + LOG.error(str(ex)) + LOG.error('Failed to import psycopg2 or pg8000 module.') + raise + + +# ----------------------------------------------------------------------------- +def check_postgresql_driver(): + try: + get_postgresql_driver_name() + return True + except Exception as ex: + LOG.debug(ex) + return False + + +# ----------------------------------------------------------------------------- +def check_sql_driver(check_postgresql): + if check_postgresql: + try: + get_postgresql_driver_name() + return True + except Exception as ex: + return False + else: + try: + try: + import pysqlite2 + except Exception as ex: + import sqlite3 + except Exception as ex: + LOG.debug(ex) + return False + return True + +# ----------------------------------------------------------------------------- +def check_clang(compiler_bin, env): + ''' + simple check if clang is available + ''' + clang_version_cmd = [compiler_bin, '--version'] + LOG.debug_analyzer(' '.join(clang_version_cmd)) + try: + res = subprocess.call(clang_version_cmd, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + if not res: + return True + else: + LOG.debug_analyzer('Failed to run: "' + ' '.join(clang_version_cmd) + '"') + return False + + except OSError as oerr: + if oerr[0] == errno.ENOENT: + LOG.error(oerr) + LOG.error('Failed to run: ' + ' '.join(clang_version_cmd) + '"') + return False + +# ----------------------------------------------------------------------------- +def check_intercept(env): + ''' + simple check if intercept (scan-build-py) is available + ''' + intercept_cmd = ['intercept-build'] + try: + res = subprocess.call(intercept_cmd, + env=env, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + + if not res: + return True + else: + LOG.debug('Failed to run: "' + ' '.join(intercept_cmd) + '"') + return False + + except OSError as oerr: + if oerr[0] == errno.ENOENT: + # not just intercept-build can be used for logging + # it is possible that another build logger is available + LOG.debug(oerr) + LOG.debug('Failed to run: ' + ' '.join(intercept_cmd) + '"') + return False Index: tools/codechecker/libcodechecker/log_parser.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/log_parser.py @@ -0,0 +1,87 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import os +import traceback + +from libcodechecker import build_action +from libcodechecker import logger +from libcodechecker import option_parser + +LOG = logger.get_new_logger('LOG PARSER') + + +# ----------------------------------------------------------------------------- +def parse_compile_commands_json(logfile): + import json + + actions = [] + filtered_build_actions = {} + + logfile.seek(0) + data = json.load(logfile) + + counter = 0 + for entry in data: + sourcefile = entry['file'] + lang = option_parser.get_language(sourcefile[sourcefile.rfind('.'):]) + + if not lang: + continue + + action = build_action.BuildAction(counter) + + command = entry['command'] + results = option_parser.parse_options(command) + + action.original_command = command + action.analyzer_options = results.compile_opts + action.lang = results.lang + action.target = results.arch + + if results.action == option_parser.ActionType.COMPILE or \ + results.action == option_parser.ActionType.LINK: + action.skip = False + + # TODO: check arch + action.directory = entry['directory'] + + action.sources = sourcefile + action.lang = lang + + # filter out duplicate compilation commands + unique_key = action.cmp_key + if filtered_build_actions.get(unique_key) is None: + filtered_build_actions[unique_key] = action + + del action + counter += 1 + + for ba_hash, ba in filtered_build_actions.iteritems(): + actions.append(ba) + + return actions + + +# ----------------------------------------------------------------------------- +def parse_log(logfilepath): + LOG.debug_analyzer('Parsing log file: ' + logfilepath) + + actions = [] + + with open(logfilepath) as logfile: + try: + actions = parse_compile_commands_json(logfile) + except (ValueError, KeyError, TypeError) as ex: + if os.stat(logfilepath).st_size == 0: + LOG.error('The compile database is empty.') + else: + LOG.error('The compile database is not valid.') + LOG.debug(traceback.format_exc()) + LOG.debug(ex) + + LOG.debug_analyzer('Parsing log file done.') + return actions Index: tools/codechecker/libcodechecker/logger.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/logger.py @@ -0,0 +1,90 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +''' + +import os +import sys +import logging + +# the logging leves can be accesses without +# importing the logging module in other modules +DEBUG = logging.DEBUG +INFO = logging.INFO +WARNING = logging.WARNING +ERROR = logging.ERROR +CRITICAL = logging.CRITICAL +NOTSET = logging.NOTSET + + +class BColors(object): + HEADER = '\033[95m' + OKBLUE = '\033[94m' + OKGREEN = '\033[92m' + WARNING = '\033[93m' + FAIL = '\033[91m' + ENDC = '\033[0m' + +# ------------------------------------------------------------------------------ +logging.DEBUG_ANALYZER = 15 +logging.addLevelName(logging.DEBUG_ANALYZER, 'DEBUG_ANALYZER') + +class CCLogger(logging.Logger): + def __init__(self, name, level=NOTSET): + return super(CCLogger, self).__init__(name, level) + + def debug_analyzer(self, msg, *args, **kwargs): + if self.isEnabledFor(logging.DEBUG_ANALYZER): + self._log(logging.DEBUG_ANALYZER, msg, args, **kwargs) + +logging.setLoggerClass(CCLogger) + +# ------------------------------------------------------------------------------ +def get_log_level(): + ''' + ''' + level = os.getenv('CODECHECKER_VERBOSE') + if level: + if level == 'debug': + return logging.DEBUG + elif level == 'debug_analyzer': + return logging.DEBUG_ANALYZER + + return logging.INFO + +# ------------------------------------------------------------------------------ +def get_new_logger(logger_name, out_stream=sys.stdout): + ''' + ''' + loglevel = get_log_level() + logger = logging.getLogger('[' + logger_name + ']') + stdout_handler = logging.StreamHandler(stream=out_stream) + + if not getattr(logger, 'handlers'): + if loglevel == logging.DEBUG: + # FIXME create a new handler to write all log messages into a file + # FIXME filter stdout messages only from analyzer, parser, + # report, sourceDep (any other?) are printed out to the stdout + logger.setLevel(logging.DEBUG) + stdout_handler.setLevel(logging.DEBUG) + format_str = '[%(process)d] <%(thread)d> - %(filename)s:%(lineno)d %(funcName)s() - %(message)s' + msg_formatter = logging.Formatter(format_str) + stdout_handler.setFormatter(msg_formatter) + logger.addHandler(stdout_handler) + elif loglevel == logging.DEBUG_ANALYZER: + logger.setLevel(logging.DEBUG_ANALYZER) + stdout_handler.setLevel(logging.DEBUG_ANALYZER) + msg_formatter = logging.Formatter('[%(levelname)s] - %(message)s') + stdout_handler.setFormatter(msg_formatter) + logger.addHandler(stdout_handler) + else: + logger.setLevel(logging.INFO) + stdout_handler.setLevel(logging.INFO) + msg_formatter = logging.Formatter('[%(levelname)s] - %(message)s') + stdout_handler.setFormatter(msg_formatter) + logger.addHandler(stdout_handler) + + return logger Index: tools/codechecker/libcodechecker/option_parser.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/option_parser.py @@ -0,0 +1,423 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +import re +import shlex + +from libcodechecker import logger + +LOG = logger.get_new_logger('OPTION PARSER') + +# TODO: make modular option handling system, +# configuring possibility from config file +# class OpenFilterPluginBase(object): +# __metaclass__ = abc.ABCMeta +# +# @abc.abstractproperty +# def compiler_name(self): +# ''' Compiler name (regex expression)''' +# return None +# +# @abc.abstactmethod +# def get_exclude_rules(self): +# ''' Remove options, x -> nothing''' +# return {} +# +# @abc.abstactmethod +# def get_modify_rules(self): +# ''' Modify options, x -> y''' +# return {} +# +# @abc.abstactmethod +# def get_add_rules(self): +# ''' Add rules,::x ''' +# return [] +# +# class DefaultFilterPlugin(OpenFilterPluginBase): +# __compiler_name = '.*' +# +# @property +# def compiler_name(self): +# return __compiler_name +# +# def get_exclude_rules(self): +# return {} +# +# def get_modify_rules(self): +# return {} +# +# def get_add_rules(self): +# return [] + +# ---------------------------------------------------------------------------- # +# Lookup tables. From ScanBuild: clang-analyzer.llvm.org/scan-build.html +# ---------------------------------------------------------------------------- # + +COMPILE_OPTION_MAP = { + '-nostdinc': 0, + '-include': 1, + '-idirafter': 1, + '-imacros': 1, + '-iprefix': 1, + '-isystem': 1, + '-iwithprefix': 1, + '-iwithprefixbefore': 1 +} + +COMPILE_OPTION_MAP_REGEX = { + '-O([1-3]|s)?$': 0, + '-std=.*': 0, + '^-f.*': 0, + '-m.*': 0, + '^-Wno-.*': 0, + '^-m(32|64)$': 0 +} + +COMPILE_OPTION_MAP_MERGED = [ + '^-iquote(.*)$', + '^-[DIU](.*)$', + '^-F(.+)$' +] + +COMPILER_LINKER_OPTION_MAP = { + '-write-strings': 0, + '-ftrapv-handler': 1, + '-mios-simulator-version-min': 0, + '-sysroot': 1, + '-stdlib': 0, + '-target': 1, + '-v': 0, + '-mmacosx-version-min': 0, + '-miphoneos-version-min': 0 +} + +LINKER_OPTION_MAP = { + '-framework': 1, + '-fobjc-link-runtime': 0 +} + +IGNORED_OPTION_MAP = { + '-MT': 1, + '-MF': 1, + '-fsyntax-only': 0, + '-save-temps': 0, + '-install_name': 1, + '-exported_symbols_list': 1, + '-current_version': 1, + '-compatibility_version': 1, + '-init': 1, + '-e': 1, + '-seg1addr': 1, + '-bundle_loader': 1, + '-multiply_defined': 1, + '-sectorder': 3, + '--param': 1, + '-u': 1, + '--serialize-diagnostics': 1 +} + +UNIQUE_OPTIONS = { + '-isysroot': 1 +} + +REPLACE_OPTIONS_MAP = { + '-mips32' : [ '-target', 'mips', '-mips32' ], + '-mips64' : [ '-target', 'mips64', '-mips64' ], + '-mpowerpc' : [ '-target', 'powerpc' ], + '-mpowerpc64' : [ '-target', 'powerpc64' ] +} + +UNKNOWN_OPTIONS_MAP_REGEX = { + '^-fcall-saved-.*': 0, + '^-fcond-mismatch': 0, + '^-fconserve-stack': 0, + '^-fcrossjumping': 0, + '^-fcse-follow-jumps': 0, + '^-fcse-skip-blocks': 0, + '^-ffixed-r2': 0, + '^-fgcse-lm': 0, + '^-fhoist-adjacent-loads': 0, + '^-findirect-inlining': 0, + '^-finline-limit.*': 0, + '^-fipa-sra': 0, + '^-fno-delete-null-pointer-checks': 0, + '^-fno-strength-reduce': 0, + '^-fno-toplevel-reorder': 0, + '^-fno-unit-at-a-time': 0, + '^-fno-var-tracking-assignments': 0, + '^-fpartial-inlining': 0, + '^-fpeephole2': 0, + '^-fregmove': 0, + '^-frename-registers': 0, + '^-freorder-functions': 0, + '^-frerun-cse-after-loop': 0, + '^-fsched-spec': 0, + '^-fthread-jumps': 0, + '^-ftree-pre': 0, + '^-ftree-switch-conversion': 0, + '^-ftree-tail-merge': 0, + '^-m(no-)?sdata.*$': 0, + '^-m(no-)?spe.*': 0, + '^-m(no-)?string$': 0, + '^-maccumulate-outgoing-args': 0, + '^-mfix-cortex-m3-ldrd$': 0, + '^-mmultiple$': 0, + '^-mthumb-interwork$': 0, + '^-mupdate$': 0 +} + +# ----------------------------------------------------------------------------- +class ActionType(object): + LINK, COMPILE, PREPROCESS, INFO = range(4) + + +# ----------------------------------------------------------------------------- +class OptionParserResult(object): + + def __init__(self): + self._action = ActionType.LINK + self._compile_opts = [] + self._link_opts = [] + self._files = [] + self._arch = '' + self._lang = None + self._output = '' + + @property + def action(self): + return self._action + + @action.setter + def action(self, value): + self._action = value + + @property + def compile_opts(self): + return self._compile_opts + + @compile_opts.setter + def compile_opts(self, value): + self._compile_opts = value + + @property + def link_opts(self): + return self._link_opts + + @link_opts.setter + def link_opts(self, value): + self._link_opts = value + + @property + def files(self): + return self._files + + @files.setter + def files(self, value): + self._files = value + + @property + def arch(self): + return self._arch + + @arch.setter + def arch(self, value): + self._arch = value + + @property + def lang(self): + return self._lang + + @lang.setter + def lang(self, value): + self._lang = value + + @property + def output(self): + return self._output + + @output.setter + def output(self, value): + self._output = value + + +# ----------------------------------------------------------------------------- +class OptionIterator(object): + + def __init__(self, args): + self._item = None + self._it = iter(args) + + def next(self): + self._item = next(self._it) + return self # ._item + + def __iter__(self): + return self + + @property + def item(self): + return self._item + + +# ----------------------------------------------------------------------------- +def arg_check(it, result): + def regex_match(string, pattern): + regexp = re.compile(pattern) + match = regexp.match(string) + return match is not None + + # Handler functions for options + def append_to_list(table, target_list, regex=False): + '''Append n item from iterator to to result[att_name] list.''' + def wrapped(value): + def append_n(size): + target_list.append(it.item) + for x in xrange(0, size): + it.next() + target_list.append(it.item) + + if regex: + for pattern in table: + if regex_match(value, pattern): + append_n(table[pattern]) + return True + elif value in table: + append_n(table[value]) + return True + return False + return wrapped + + def append_merged_to_list(table, target_list): + ''' Append one or two item to the list. + 1: if there is no space between two option. + 2: otherwise. + ''' + def wrapped(value): + for pattern in table: + match = re.match(pattern, value) + if match is not None: + tmp = it.item + if match.group(1) == '': + it.next() + tmp = tmp + it.item + target_list.append(tmp) + return True + return False + return wrapped + + def append_to_list_from_file(arg, target_list): + '''Append items from file to to result[att_name] list.''' + def wrapped(value): + if value == arg: + it.next() + with open(it.item) as file: + for line in file: + target_list.append(line.strip()) + return True + return False + return wrapped + + def append_replacement_to_list(table, target_list, regex=False): + '''Append replacement items from table to to result[att_name] list.''' + def wrapped(value): + def append_replacement(items): + for item in items: + target_list.append(item) + it.next() + + if regex: + for pattern in table: + if regex_match(value, pattern): + append_replacement(table[pattern]) + return True + elif value in table: + append_replacement(table[value]) + return True + return False + return wrapped + + def set_attr(arg, attr_name, attr_value=None, regex=None): + '''Set an attr value. If no value given then read next from iterator.''' + def wrapped(value): + if (regex and regex_match(value, arg)) or value == arg: + tmp = attr_value + if attr_value is None: + it.next() + tmp = it.item + attr_name = tmp + return True + return False + return wrapped + + def skip(table, regex=False): + '''Skip n item in iterator.''' + def wrapped(value): + def skip_n(size): + for x in xrange(0, size): + it.next() + + if regex: + for pattern in table: + if regex_match(value, pattern): + table[pattern] + return True + elif value in table: + skip_n(table[value]) + return True + return False + return wrapped + + # Defines handler functions for tables and single options + arg_collection = [ + append_replacement_to_list(REPLACE_OPTIONS_MAP, result.compile_opts), + skip(UNKNOWN_OPTIONS_MAP_REGEX, True), + append_to_list(COMPILE_OPTION_MAP, result.compile_opts), + append_to_list(COMPILER_LINKER_OPTION_MAP, result.compile_opts), + append_to_list(COMPILE_OPTION_MAP_REGEX, result.compile_opts, True), + append_merged_to_list(COMPILE_OPTION_MAP_MERGED, result.compile_opts), + set_attr('-x', result.lang), + set_attr('-o', result.output), + set_attr('-arch', result.arch), + set_attr('-c', result.action, ActionType.COMPILE), + set_attr('^-(E|MM?)$', result.action, ActionType.PREPROCESS, True), + set_attr('-print-prog-name', result.action, ActionType.INFO), + skip(LINKER_OPTION_MAP), + skip(IGNORED_OPTION_MAP), + skip(UNIQUE_OPTIONS), + append_to_list_from_file('-filelist', result.files), + append_to_list({'^[^-].+': 0}, result.files, True)] + + return any((collection(it.item) for collection in arg_collection)) + + +# ----------------------------------------------------------------------------- +def parse_options(args): + '''Requires a full compile command with the compiler, not only arguments.''' + + # keep " characters + args = args.replace('"', '\\"') + + result_map = OptionParserResult() + for it in OptionIterator(shlex.split(args)[1:]): + arg_check(it, result_map) # TODO: do sth at False result, actually skip + + return result_map + + +# ----------------------------------------------------------------------------- +def get_language(extension): + mapping = {'.c': 'c', + '.cp': 'c++', + '.cpp': 'c++', + '.cxx': 'c++', + '.txx': 'c++', + '.cc': 'c++', + '.C': 'c++', + '.ii': 'c++', + '.m': 'objective-c', + '.mm': 'objective-c++'} + lang = mapping.get(extension) if extension in mapping else None + return lang Index: tools/codechecker/libcodechecker/orm_model.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/orm_model.py @@ -0,0 +1,284 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +ORM model. +''' +from __future__ import print_function +from __future__ import unicode_literals + +from sqlalchemy import * +from sqlalchemy.orm import * +from sqlalchemy.sql.expression import true +from sqlalchemy.ext.declarative import declarative_base + +from datetime import datetime +from math import ceil + +CC_META = MetaData(naming_convention={ + "ix": 'ix_%(column_0_label)s', + "uq": "uq_%(table_name)s_%(column_0_name)s", + "ck": "ck_%(table_name)s_%(column_0_name)s", + "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s", + "pk": "pk_%(table_name)s" +}) + +# Create base class for ORM classes +Base = declarative_base(metadata=CC_META) + +# Start of ORM classes + +class DBVersion(Base): + __tablename__ = 'db_version' + # TODO: constraint, only one line in this table + major = Column(Integer, primary_key=True) + minor = Column(Integer, primary_key=True) + + def __init__(self, major, minor): + self.major = major + self.minor = minor + + +class Run(Base): + __tablename__ = 'runs' + + __table_args__ = ( + UniqueConstraint('name'), + ) + + id = Column(Integer, autoincrement=True, primary_key=True) + date = Column(DateTime) + duration = Column(Integer) # Seconds, -1 if unfinished. + name = Column(String) + version = Column(String) + command = Column(String) + inc_count = Column(Integer) + can_delete = Column(Boolean, nullable=False, server_default=true(), default=True) + + # Relationships (One to Many). + configlist = relationship('Config', cascade="all, delete-orphan", passive_deletes=True) + buildactionlist = relationship('BuildAction', cascade="all, delete-orphan", passive_deletes=True) + + def __init__(self, name, version, command): + self.date, self.name, self.version, self.command = \ + datetime.now(), name, version, command + self.duration = -1 + self.inc_count = 0 + + def mark_finished(self): + self.duration = ceil((datetime.now() - self.date).total_seconds()) + + +class Config(Base): + __tablename__ = 'configs' + + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), primary_key=True) + checker_name = Column(String, primary_key=True) + attribute = Column(String, primary_key=True) + value = Column(String, primary_key=True) + + def __init__(self, run_id, checker_name, attribute, value): + self.attribute, self.value = attribute, value + self.checker_name, self.run_id = checker_name, run_id + +class File(Base): + __tablename__ = 'files' + + id = Column(Integer, autoincrement=True, primary_key=True) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + filepath = Column(String) + content = Column(Binary) + inc_count = Column(Integer) + + def __init__(self, run_id, filepath): + self.run_id, self.filepath = run_id, filepath + self.inc_count = 0 + + def addContent(self, content): + self.content = content + + +class BuildAction(Base): + __tablename__ = 'build_actions' + + id = Column(Integer, autoincrement=True, primary_key=True) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + build_cmd = Column(String) + analyzer_type = Column(String, nullable=False) + analyzed_source_file = Column(String, nullable=False) + check_cmd = Column(String) + # No failure if the text is empty. + failure_txt = Column(String) + date = Column(DateTime) + + # Seconds, -1 if unfinished. + duration = Column(Integer) + + def __init__(self, run_id, build_cmd, check_cmd, analyzer_type, analyzed_source_file): + self.run_id, self.build_cmd, self.check_cmd, self.failure_txt = \ + run_id, build_cmd, check_cmd, '' + self.date = datetime.now() + self.analyzer_type = analyzer_type + self.analyzed_source_file = analyzed_source_file + self.duration = -1 + + def mark_finished(self, failure_txt): + self.failure_txt = failure_txt + self.duration = (datetime.now() - self.date).total_seconds() + + +class BugPathEvent(Base): + __tablename__ = 'bug_path_events' + + id = Column(Integer, autoincrement=True, primary_key=True) + line_begin = Column(Integer) + col_begin = Column(Integer) + line_end = Column(Integer) + col_end = Column(Integer) + msg = Column(String) + file_id = Column(Integer, ForeignKey('files.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + + next = Column(Integer) + prev = Column(Integer) + + def __init__(self, line_begin, col_begin, line_end, col_end, msg, file_id): + self.line_begin, self.col_begin, self.line_end, self.col_end, self.msg = \ + line_begin, col_begin, line_end, col_end, msg + self.file_id = file_id + + def addPrev(self, prev): + self.prev = prev + + def addNext(self, next): + self.next = next + + def isLast(self): + return self.next is None + + def isFirst(self): + return self.prev is None + + + +class BugReportPoint(Base): + __tablename__ = 'bug_report_points' + + id = Column(Integer, autoincrement=True, primary_key=True) + line_begin = Column(Integer) + col_begin = Column(Integer) + line_end = Column(Integer) + col_end = Column(Integer) + file_id = Column(Integer, ForeignKey('files.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + + # TODO: Add check, the value must be an existing id or null. + # Be careful when inserting. + next = Column(Integer) + + def __init__(self, line_begin, col_begin, line_end, col_end, file_id): + self.line_begin, self.col_begin, self.line_end, self.col_end = \ + line_begin, col_begin, line_end, col_end + self.file_id = file_id + + def addNext(self, next): + self.next = next + + def isLast(self): + return self.next is None + + +class Report(Base): + __tablename__ = 'reports' + + id = Column(Integer, autoincrement=True, primary_key=True) + file_id = Column(Integer, ForeignKey('files.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + bug_id = Column(String, index = True) + checker_id = Column(String) + checker_cat = Column(String) + bug_type = Column(String) + severity = Column(Integer) + + # TODO: multiple messages to multiple source locations? + checker_message = Column(String) + start_bugpoint = Column(Integer, ForeignKey('bug_report_points.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE')) + + start_bugevent = Column(Integer, ForeignKey('bug_path_events.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + end_bugevent = Column(Integer, ForeignKey('bug_path_events.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), index = True) + suppressed = Column(Boolean) + + # Cascade delete might remove rows SQLAlchemy warns about this + # to remove warnings about already deleted items set this to False. + __mapper_args__ = { + 'confirm_deleted_rows' : False + } + + + # Priority/severity etc... + def __init__(self, run_id, bug_id, file_id, checker_message, start_bugpoint, start_bugevent, end_bugevent, checker_id, checker_cat, bug_type, severity, suppressed): + self.run_id = run_id + self.file_id = file_id + self.bug_id, self.checker_message = bug_id, checker_message + self.start_bugpoint = start_bugpoint + self.start_bugevent = start_bugevent + self.end_bugevent = end_bugevent + self.severity = severity + self.checker_id, self.checker_cat, self.bug_type = checker_id, checker_cat, bug_type + self.suppressed = suppressed + +class ReportsToBuildActions(Base): + __tablename__ = 'reports_to_build_actions' + + report_id = Column(Integer, ForeignKey('reports.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), primary_key=True) + build_action_id = Column( + Integer, ForeignKey('build_actions.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), primary_key=True) + + def __init__(self, report_id, build_action_id): + self.report_id = report_id + self.build_action_id = build_action_id + + +class SuppressBug(Base): + __tablename__ = 'suppress_bug' + + id = Column(Integer, autoincrement=True, primary_key=True) + hash = Column(String, nullable=False) + file_name = Column(String) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), nullable=False) + comment = Column(Binary) + + def __init__(self, run_id, hash, file_name, comment): + self.hash, self.run_id = hash, run_id + self.comment = comment + self.file_name = file_name + + +class SkipPath(Base): + __tablename__ = 'skip_path' + + id = Column(Integer, autoincrement=True, primary_key=True) + path = Column(String) + run_id = Column(Integer, ForeignKey('runs.id', deferrable = True, initially = "DEFERRED", ondelete='CASCADE'), nullable=False) + comment = Column(Binary) + + def __init__(self, run_id, path, comment): + self.path = path + self.run_id = run_id + self.comment = comment + +# End of ORM classes. + + +def CreateSchema(engine): + """ Creates the schema if it does not exists. + Do not check version or do migration yet. """ + Base.metadata.create_all(engine) + + +def CreateSession(engine): + """ Creates a scoped session factory that can act like a session. + The factory uses a thread_local registry, so every thread have + its own session. """ + SessionFactory = scoped_session(sessionmaker(bind=engine)) + return SessionFactory Index: tools/codechecker/libcodechecker/pgpass.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/pgpass.py @@ -0,0 +1,90 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +Parser for PostgreSQL libpq password file. +''' + +def _match_field(line, field): + ''' + Return the remaining part of the line in case of matching field, + otherwise it returns None + ''' + + if line is None or len(line) < 2: + return None + + if line[0] == '*' and line[1] == ':': + # match (star) + return line[2:] + + escaped = False + while len(line) > 0: + if not escaped and line[0] == '\\': + line = line[1:] + escaped = True + + if len(line) == 0: + return None + + if not escaped and line[0] == ':' and len(field) == 0: + # match + return line[1:] + + escaped = False + if len(field) == 0: + return None + elif field[0] == line[0]: + line = line[1:] + field = field[1:] + else: + return None + return None + +def _match_line(line, hostname, port, database, username): + ''' + Tries to match the given line to the given hostname, port, database, and username. + + Returns non None on match. + ''' + + pw = _match_field(line, hostname) + pw = _match_field(pw, port) + pw = _match_field(pw, database) + pw = _match_field(pw, username) + if pw is None: + return pw + + # the password is still escaped + escaped = False + password = '' + for c in pw: + if not escaped and c == '\\': + escaped = True + continue + + escaped = False + password += c + return password + +def get_password_from_file(passfile_path, hostname, port, database, username): + ''' + Parser for PostgreSQL libpq password file. + + Returns None on no matching entry, otherwise it returns the password. + + For file format see http://www.postgresql.org/docs/current/static/libpq-pgpass.html + ''' + + if len(hostname) == 0 or len(port) == 0 or len(database) == 0 or len(username) == 0: + return None + + with open(passfile_path, 'r') as passfile: + for line in passfile: + pw = _match_line(line.strip(), hostname, port, database, username) + if pw: + return pw + + return None Index: tools/codechecker/libcodechecker/plist_helper.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/plist_helper.py @@ -0,0 +1,113 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' This file contains some workaround for codechecker +to work with older clang versions. It is for demonstration purposes only. +The names, hashes will change after switching to a newer clang version. +''' + +import hashlib +import linecache +import os +import re + +from libcodechecker import logger + +LOG = logger.get_new_logger('PLIST_HELPER') + + +def gen_bug_hash(bug): + line_content = linecache.getline(bug.file_path, bug.from_line) + if line_content == '' and not os.path.isfile(bug.file_path): + LOG.debug('%s does not exists!' % bug.file_path) + + file_name = os.path.basename(bug.file_path) + l = [file_name, bug.checker_name, bug.msg, line_content, + str(bug.from_col), str(bug.until_col)] + for p in bug.paths(): + l.append(str(p.start_pos.col)) + l.append(str(p.end_pos.col)) + string_to_hash = '|||'.join(l) + return hashlib.md5(string_to_hash.encode()).hexdigest() + + +def levenshtein(a, b): # http://hetland.org/coding/python/levenshtein.py + "Calculates the Levenshtein distance between a and b." + n, m = len(a), len(b) + if n > m: + # Make sure n <= m, to use O(min(n,m)) space + a, b = b, a + n, m = m, n + + current = range(n+1) + for i in range(1, m+1): + previous, current = current, [i]+[0]*n + for j in range(1, n+1): + add, delete = previous[j]+1, current[j-1]+1 + change = previous[j-1] + if a[j-1] != b[i-1]: + change = change + 1 + current[j] = min(add, delete, change) + + return current[n] + + +def get_check_name(current_msg): + # clean message from variable and class name + clean_msg = re.sub(r"'.*?'", '', current_msg) + + closest_msg = '' + min_dist = len(clean_msg) // 4 + + for msg in checker_message_map.keys(): + tmp_dist = levenshtein(clean_msg, msg) + if tmp_dist < min_dist: + closest_msg = msg + min_dist = tmp_dist + + return checker_message_map[closest_msg] + + +# this map needs extending +checker_message_map = \ + { + "": "NOT FOUND", + "Access out-of-bound array element (buffer overflow)": "alpha.security.ArrayBound", + "Address of stack memory associated with local variable returned to caller": "core.StackAddressEscape", + "Argument to is uninitialized": "core.CallAndMessage", + "Argument to free() is the address of the global variable , which is not memory allocated by malloc()": "alpha.unix.MallocWithAnnotations", + "Argument to free() is the address of the local variable , which is not memory allocated by malloc()": "alpha.unix.MallocWithAnnotations", + "Assigned value is garbage or undefined": "core.uninitialized.Assign", + "Attempt to free released memory": "alpha.unix.MallocWithAnnotations", + "Branch condition evaluates to a garbage value": "core.uninitialized.Branch", + "Call to function is extremely insecure as it can always result in a buffer overflow": "security.insecureAPI.gets", + "Call to function is insecure as it always creates or uses insecure temporary file. Use instead": "security.insecureAPI.mktemp", + "Call to function is insecure as it does not provide bounding of the memory buffer. Replace unbounded copy functions with analogous functions that support length arguments such as . CWE-119": "security.insecureAPI.strcpy", + "Cast a region whose size is not a multiple of the destination type size": "alpha.core.CastSize", + "Dereference of null pointer (loaded from variable )": "core.NullDereference", + "Dereference of undefined pointer value": "core.NullDereference", + "Division by zero": "core.DivideZero", + "Function call argument is a pointer to uninitialized value": "core.CallAndMessage", + "Function call argument is an uninitialized value": "core.CallAndMessage", + "identical expressions on both sides of logical operator": "alpha.core.IdenticalExpr", + "Memory allocated by should be deallocated by , not ": "unix.MismatchedDeallocator", + "Memory allocated by should be deallocated by , not free()": "unix.MismatchedDeallocator", + "No call of chdir(\"/\") immediately after chroot": "alpha.unix.Chroot", + "Opened File never closed. Potential Resource leak": "alpha.unix.Stream", + "Potential leak of memory pointed to by ": "alpha.unix.MallocWithAnnotations", + "Potential leak of memory pointed to by ": "cplusplus.NewDelete", + "Potential memory leak": "cplusplus.NewDelete", + "Result of is converted to a pointer of type , which is incompatible with sizeof operand type ": "unix.MallocSizeof", + "Size argument is greater than the length of the destination buffer": "alpha.unix.cstring.OutOfBounds", + "The code calls sizeof() on a pointer type. This can produce an unexpected result": "alpha.core.SizeofPtr", + "the computation of the size of the memory allocation may overflow": "alpha.security.MallocOverflow", + "The left operand of is a garbage value": "core.UndefinedBinaryOperatorResult", + "The right operand of is a garbage value": "core.UndefinedBinaryOperatorResult", + "This statement is never executed": "alpha.deadcode.UnreachableCode", + "Undefined or garbage value returned to caller": "core.uninitialized.UndefReturn", + "Use of memory after it is freed": "alpha.unix.MallocWithAnnotations", + "Value stored to during its initialization is never read": "deadcode.DeadStores", + "Value stored to is never read": "deadcode.DeadStores" +} Index: tools/codechecker/libcodechecker/plist_parser.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/plist_parser.py @@ -0,0 +1,171 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +'''''' + +# FIXME: This file contains some workarounds. +# Remove them as soon as a proper clang version comes out. + +import plistlib +from xml.parsers.expat import ExpatError + +import plist_helper +from libcodechecker import logger + +LOG = logger.get_new_logger('PLIST_PARSER') + + +class GenericEquality(object): + + def __eq__(self, other): + return (isinstance(other, self.__class__) and + self.__dict__ == other.__dict__) + + def __ne__(self, other): + return not self.__eq__(other) + + +# ----------------------------------------------------------------------------- +class Position(GenericEquality): + + '''Represent a postion.''' + + def __init__(self, x, y, f): + self.line = x + self.col = y + self.file_path = f + + +# ----------------------------------------------------------------------------- +class Range(GenericEquality): + + '''Represent a location in the bug path.''' + + def __init__(self, start_pos, end_pos, msg=''): + self.start_pos = start_pos + self.end_pos = end_pos + self.msg = msg + + +# ----------------------------------------------------------------------------- +def make_position(pos_map, files): + return Position(pos_map.line, pos_map.col, files[pos_map.file]) + + +# ----------------------------------------------------------------------------- +def make_range(array, files): + if len(array) == 2: + start = make_position(array[0], files) + end = make_position(array[1], files) + return Range(start, end) + + +# ----------------------------------------------------------------------------- +class Bug(object): + + '''The bug with all information, it include bugpath too.''' + + def __init__(self, file, from_pos, until_pos=None, msg=None, category=None, + type=None, hash_value=''): + self.file_path = file + self.msg = msg + self._paths = [] + self._events = [] + self.hash_value = hash_value + + if not until_pos: + until_pos = from_pos + + (self.from_line, self.from_col) = from_pos + (self.until_line, self.until_col) = until_pos + + def paths(self): + return self._paths + + def events(self): + return self._events + + def add_to_path(self, new_range): + self._paths.append(new_range) + + def add_to_events(self, new_range): + self._events.append(new_range) + + def get_last_path(self): + return self._paths[-1] if len(self._paths) > 0 else None + + def get_last_event(self): + return self._events[-1] if len(self._events) > 0 else None + + +# ----------------------------------------------------------------------------- +def parse_plist(path): + """ + parse the plist file + """ + bugs = [] + files = [] + try: + plist = plistlib.readPlist(path) + + files = plist['files'] + + for diag in plist['diagnostics']: + current = Bug(files[diag['location']['file']], + (diag['location']['line'], diag['location']['col'])) + + for item in diag['path']: + if item['kind'] == 'event': + message = item['message'] + if 'ranges' in item: + for arr in item['ranges']: + source_range = make_range(arr, files) + source_range.msg = message + current.add_to_events(source_range) + else: + location = make_position(item['location'], files) + source_range = Range(location, location, message) + source_range.msg = message + current.add_to_events(source_range) + + elif item['kind'] == 'control': + for edge in item['edges']: + start = make_range(edge.start, files) + end = make_range(edge.end, files) + + if start != current.get_last_path(): + current.add_to_path(start) + + current.add_to_path(end) + + current.msg = diag['description'] + current.category = diag['category'] + current.type = diag['type'] + + try: + current.checker_name = diag['check_name'] + except KeyError as kerr: + LOG.debug("Check name wasn't found in the plist file. " + 'Read the user guide!') + current.checker_name = plist_helper.get_check_name(current.msg) + LOG.debug('Guessed check name: ' + current.checker_name) + + try: + current.hash_value = diag['issue_hash_content_of_line_in_context'] + except KeyError as kerr: + # hash was not found + # generate some hash for older clang versions + LOG.debug(kerr) + LOG.debug("Hash value wasn't found in the plist file. " + 'Read the user guide!') + current.hash_value = plist_helper.gen_bug_hash(current) + + bugs.append(current) + + except ExpatError as err: + LOG.debug('Failed to process plist file: ' + path) + LOG.debug(err) + finally: + return files, bugs Index: tools/codechecker/libcodechecker/skiplist_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/skiplist_handler.py @@ -0,0 +1,65 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +""" +""" + +import re +import fnmatch +from libcodechecker import logger + +LOG = logger.get_new_logger('SKIPLIST_HANDLER') + +class SkipListHandler(object): + """ + Skiplist file format: + + -/skip/all/source/in/directory* + -/do/not/check/this.file + +/dir/check.this.file + -/dir/* + + """ + + def __init__(self, skip_file): + """ + read up the skip file + """ + self.__skip = [] + + skip_file_content = [] + + with open(skip_file, 'r') as skip_file: + skip_file_content = [line.strip() for line in skip_file if line.strip() != ''] + + for line in skip_file_content: + if len(line) < 2 or line[0] not in ['-', '+']: + LOG.warning("Skipping malformed skipfile pattern: " + line) + continue + rexpr = re.compile(fnmatch.translate(line[1:].strip() + '*')) + self.__skip.append((line, rexpr)) + + def should_skip(self, source): + """ + check if the given source sourld be skipped + Should the analyzer skip the given source file? + """ + + for line, rexpr in self.__skip: + if rexpr.match(source): + sign = line[0] + return sign == '-' + return False + + def get_skiplist(self): + """ + Read skip file and return with its content in a list. + """ + + skiplist_with_comment = {} + for line, rexpr in self.__skip: + skiplist_with_comment[line] = '' + + return skiplist_with_comment Index: tools/codechecker/libcodechecker/storage_server/__init__.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/storage_server/__init__.py @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. Index: tools/codechecker/libcodechecker/storage_server/report_server.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/storage_server/report_server.py @@ -0,0 +1,640 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +from __future__ import print_function +from __future__ import unicode_literals + +import sys +import os +import datetime +import socket +import errno +import ntpath + +import sqlalchemy + +from thrift.transport import TSocket +from thrift.transport import TTransport +from thrift.protocol import TBinaryProtocol +from thrift.server import TServer + +from codechecker_gen.DBThriftAPI import CheckerReport +from codechecker_gen.DBThriftAPI.ttypes import * +import shared + +from libcodechecker.db_model.orm_model import * + +from libcodechecker import logger +from libcodechecker import decorators +from libcodechecker import database_handler + +LOG = logger.get_new_logger('CC SERVER') + +if os.environ.get('CODECHECKER_ALCHEMY_LOG') is not None: + import logging + + logging.basicConfig() + logging.getLogger('sqlalchemy.engine').setLevel(logging.DEBUG) + logging.getLogger('sqlalchemy.orm').setLevel(logging.DEBUG) + +class CheckerReportHandler(object): + ''' + Class to handle requests from the codechecker script to store run + information to the database. + ''' + + def __sequence_deleter(self, table, first_id): + '''Delete points of sequnce in a general way.''' + next_id = first_id + while next_id: + item = self.session.query(table).get(next_id) + if item: + next_id = item.next + self.session.delete(item) + else: + break + + def __del_source_file_for_report(self, run_id, report_id, report_file_id): + ''' + Delete the stored file if there are no report references to it + in the database. + ''' + report_reference_to_file = self.session.query(Report) \ + .filter( + and_(Report.run_id == run_id, + Report.file_id == report_file_id, + Report.id != report_id)) + rep_ref_count = report_reference_to_file.count() + if rep_ref_count == 0: + LOG.debug("No other references to the source file \n id: " + + str(report_file_id) + " can be deleted.") + # There are no other references to the file, it can be deleted. + self.session.query(File).filter(File.id == report_file_id)\ + .delete() + return rep_ref_count + + def __del_buildaction_results(self, build_action_id, run_id): + """ + Delete the build action and related analysis results from the database. + + Report entry will be deleted by ReportsToBuildActions cascade delete. + """ + LOG.debug("Cleaning old buildactions") + + try: + rep_to_ba = self.session.query(ReportsToBuildActions) \ + .filter(ReportsToBuildActions.build_action_id == + build_action_id) + + reports_to_delete = [r.report_id for r in rep_to_ba] + + LOG.debug("Trying to delete reports belonging to the buildaction:") + LOG.debug(reports_to_delete) + + for report_id in reports_to_delete: + # Check if there is another reference to this report from + # other buildactions. + other_reference = self.session.query(ReportsToBuildActions) \ + .filter( + and_(ReportsToBuildActions.report_id == report_id, + ReportsToBuildActions.build_action_id != build_action_id)) + + LOG.debug("Checking report id:" + str(report_id)) + + LOG.debug("Report id " + str(report_id) + + " reference count: " + + str(other_reference.count())) + + if other_reference.count() == 0: + # There is no other reference, data related to the report + # can be deleted. + report = self.session.query(Report).get(report_id) + + LOG.debug("Removing bug path events") + self.__sequence_deleter(BugPathEvent, report.start_bugevent) + LOG.debug("Removing bug report points") + self.__sequence_deleter(BugReportPoint, report.start_bugpoint) + + if self.__del_source_file_for_report(run_id, report.id, report.file_id): + LOG.debug("Stored source file needs to be kept, there is reference to it from another report.") + # report needs to be deleted if there is no reference the + # file cascade delete will remove it + # else manual cleanup is needed + self.session.delete(report) + + self.session.query(BuildAction).filter(BuildAction.id == build_action_id)\ + .delete() + + self.session.query(ReportsToBuildActions).filter( + ReportsToBuildActions.build_action_id == build_action_id).delete() + + except Exception as ex: + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.GENERAL, + str(ex)) + + @decorators.catch_sqlalchemy + def addCheckerRun(self, command, name, version, force): + ''' + Store checker run related data to the database. + By default updates the results if name already exists. + Using the force flag removes existing analysis results for a run. + ''' + run = self.session.query(Run).filter(Run.name == name).first() + if run and force: + # Clean already collected results. + if not run.can_delete: + # Deletion is already in progress. + msg = "Can't delete " + str(run.id) + LOG.debug(msg) + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.DATABASE, + msg) + + LOG.info('Removing previous analisys results ...') + self.session.delete(run) + self.session.commit() + + checker_run = Run(name, version, command) + self.session.add(checker_run) + self.session.commit() + return checker_run.id + + elif run: + # There is already a run, update the results. + run.date = datetime.now() + # Increment update counter. + run.inc_count += 1 + self.session.commit() + return run.id + else: + # There is no run create new. + checker_run = Run(name, version, command) + self.session.add(checker_run) + self.session.commit() + return checker_run.id + + @decorators.catch_sqlalchemy + def finishCheckerRun(self, run_id): + ''' + ''' + run = self.session.query(Run).get(run_id) + if not run: + return False + + run.mark_finished() + self.session.commit() + return True + + @decorators.catch_sqlalchemy + def replaceConfigInfo(self, run_id, config_values): + ''' + Removes all the previously stored config informations + and stores the new values. + ''' + count = self.session.query(Config) \ + .filter(Config.run_id == run_id) \ + .delete() + LOG.debug('Config: ' + str(count) + ' removed item.') + + configs = [Config( + run_id, info.checker_name, info.attribute, info.value) for + info in config_values] + self.session.bulk_save_objects(configs) + self.session.commit() + return True + + @decorators.catch_sqlalchemy + def addBuildAction(self, + run_id, + build_cmd, + check_cmd, + analyzer_type, + analyzed_source_file): + ''' + ''' + try: + + build_actions = \ + self.session.query(BuildAction) \ + .filter(and_(BuildAction.run_id == run_id, + BuildAction.build_cmd == build_cmd, + or_( + and_(BuildAction.analyzer_type == analyzer_type, + BuildAction.analyzed_source_file == analyzed_source_file), + and_(BuildAction.analyzer_type == "", + BuildAction.analyzed_source_file == "") + )))\ + .all() + + + if build_actions: + # Delete the already stored buildaction and analysis results. + for build_action in build_actions: + + self.__del_buildaction_results(build_action.id, run_id) + + self.session.commit() + + action = BuildAction(run_id, + build_cmd, + check_cmd, + analyzer_type, + analyzed_source_file) + self.session.add(action) + self.session.commit() + + except Exception as ex: + LOG.error(ex) + raise + + return action.id + + @decorators.catch_sqlalchemy + def finishBuildAction(self, action_id, failure): + ''' + ''' + action = self.session.query(BuildAction).get(action_id) + if action is None: + # TODO: if file is not needed update reportstobuildactions. + return False + + action.mark_finished(failure) + self.session.commit() + return True + + @decorators.catch_sqlalchemy + def needFileContent(self, run_id, filepath): + ''' + ''' + try: + f = self.session.query(File) \ + .filter(and_(File.run_id == run_id, + File.filepath == filepath)) \ + .one_or_none() + except Exception as ex: + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.GENERAL, + str(ex)) + + run_inc_count = self.session.query(Run).get(run_id).inc_count + needed = False + if not f: + needed = True + f = File(run_id, filepath) + self.session.add(f) + self.session.commit() + elif f.inc_count < run_inc_count: + needed = True + f.inc_count = run_inc_count + return NeedFileResult(needed, f.id) + + @decorators.catch_sqlalchemy + def addFileContent(self, id, content): + ''' + ''' + f = self.session.query(File).get(id) + assert f is not None + f.addContent(content) + return True + + def __is_same_event_path(self, start_bugevent_id, events): + ''' + Checks if the given event path is the same as the one in the + events argument. + ''' + try: + # There should be at least one bug event. + point2 = self.session.query(BugPathEvent).get(start_bugevent_id) + + for point1 in events: + if point1.startLine != point2.line_begin or \ + point1.startCol != point2.col_begin or \ + point1.endLine != point2.line_end or \ + point1.endCol != point2.col_end or \ + point1.msg != point2.msg or \ + point1.fileId != point2.file_id: + return False + + if point2.next is None: + return point1 == events[-1] + + point2 = self.session.query(BugPathEvent).get(point2.next) + + except Exception as ex: + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.GENERAL, + str(ex)) + + def storeReportInfo(self, + action, + file_id, + bug_hash, + msg, + bugpath, + events, + checker_id, + checker_cat, + bug_type, + severity, + suppressed = False): + ''' + ''' + try: + path_ids = self.storeBugPath(bugpath) + event_ids = self.storeBugEvents(events) + path_start = path_ids[0].id if len(path_ids) > 0 else None + + source_file = self.session.query(File).get(file_id) + source_file_path, source_file_name = ntpath.split(source_file.filepath) + + # Old suppress format did not contain file name. + suppressed = self.session.query(SuppressBug) \ + .filter( + and_(SuppressBug.run_id == action.run_id, + SuppressBug.hash == bug_hash, + or_(SuppressBug.file_name == source_file_name, + SuppressBug.file_name == u''))) \ + .count() > 0 + + report = Report(action.run_id, + bug_hash, + file_id, + msg, + path_start, + event_ids[0].id, + event_ids[-1].id, + checker_id, + checker_cat, + bug_type, + severity, + suppressed) + + self.session.add(report) + self.session.commit() + # Commit required to get the ID of the newly added report. + reportToActions = ReportsToBuildActions(report.id, action.id) + self.session.add(reportToActions) + # Avoid data loss for duplicate keys. + self.session.commit() + return report.id + except Exception as ex: + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.GENERAL, + str(ex)) + + @decorators.catch_sqlalchemy + def addReport(self, + build_action_id, + file_id, + bug_hash, + msg, + bugpath, + events, + checker_id, + checker_cat, + bug_type, + severity, + suppress): + ''' + ''' + try: + action = self.session.query(BuildAction).get(build_action_id) + assert action is not None + + # TODO: perfomance issues when executing the following query on large + # databaseses? + reports = self.session.query(self.report_ident) \ + .filter(and_(self.report_ident.c.bug_id == bug_hash, + self.report_ident.c.run_id == action.run_id)) + try: + # Check for duplicates by bug hash. + if reports.count() != 0: + for possib_dup in reports: + # It's a duplicate or a hash clash. Check checker name, + # file id, and position. + dup_report_obj = self.session.query(Report).get( + possib_dup.report_ident.id) + if dup_report_obj and dup_report_obj.checker_id == checker_id and \ + dup_report_obj.file_id == file_id and \ + self.__is_same_event_path(dup_report_obj.start_bugevent, events): + # It's a duplicate. + rtp = self.session.query(ReportsToBuildActions) \ + .get((dup_report_obj.id, + action.id)) + if not rtp: + reportToActions = ReportsToBuildActions( + dup_report_obj.id, action.id) + self.session.add(reportToActions) + self.session.commit() + return dup_report_obj.id + + return self.storeReportInfo(action, + file_id, + bug_hash, + msg, + bugpath, + events, + checker_id, + checker_cat, + bug_type, + severity, + suppress) + + except sqlalchemy.exc.IntegrityError as ex: + self.session.rollback() + + reports = self.session.query(self.report_ident) \ + .filter(and_(self.report_ident.c.bug_id == bug_hash, + self.report_ident.c.run_id == action.run_id)) + if reports.count() != 0: + return reports.first().report_ident.id + else: + raise + except Exception as ex: + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.GENERAL, + str(ex)) + + def storeBugEvents(self, bugevents): + ''' + ''' + events = [] + for event in bugevents: + bpe = BugPathEvent(event.startLine, + event.startCol, + event.endLine, + event.endCol, + event.msg, + event.fileId) + self.session.add(bpe) + events.append(bpe) + + self.session.flush() + + if len(events) > 1: + for i in xrange(len(events)-1): + events[i].addNext(events[i+1].id) + events[i+1].addPrev(events[i].id) + events[-1].addPrev(events[-2].id) + return events + + def storeBugPath(self, bugpath): + paths = [] + for i in xrange(len(bugpath)): + brp = BugReportPoint(bugpath[i].startLine, + bugpath[i].startCol, + bugpath[i].endLine, + bugpath[i].endCol, + bugpath[i].fileId) + self.session.add(brp) + paths.append(brp) + + self.session.flush() + + for i in xrange(len(paths)-1): + paths[i].addNext(paths[i+1].id) + + return paths + + @decorators.catch_sqlalchemy + def addSuppressBug(self, run_id, bugs_to_suppress): + ''' + Supppress multiple bugs for a run. This can be used before storing + the suppress file content. + ''' + + try: + suppressList = [] + for bug_to_suppress in bugs_to_suppress: + suppress_bug = SuppressBug(run_id, + bug_to_suppress.bug_hash, + bug_to_suppress.file_name, + bug_to_suppress.comment) + suppressList.append(suppress_bug) + + self.session.bulk_save_objects(suppressList) + self.session.commit() + + except Exception as ex: + LOG.error(str(ex)) + return False + + return True + + + @decorators.catch_sqlalchemy + def cleanSuppressData(self, run_id): + ''' + Clean the suppress bug entries for a run + and remove suppressed flags for the suppressed reports. + Only the database is modified. + ''' + + try: + count = self.session.query(SuppressBug) \ + .filter(SuppressBug.run_id == run_id) \ + .delete() + LOG.debug('Cleaning previous suppress entries from the database. ' + + str(count) + ' removed items.') + + reports = self.session.query(Report) \ + .filter(and_(Report.run_id == run_id, + Report.suppressed == True)) \ + .all() + + for report in reports: + report.suppressed = False + + self.session.commit() + + except Exception as ex: + LOG.error(str(ex)) + return False + + return True + + + @decorators.catch_sqlalchemy + def addSkipPath(self, run_id, paths): + ''' + ''' + count = self.session.query(SkipPath) \ + .filter(SkipPath.run_id == run_id) \ + .delete() + LOG.debug('SkipPath: ' + str(count) + ' removed item.') + + skipPathList = [] + for path, comment in paths.items(): + skipPath = SkipPath(run_id, path, comment) + skipPathList.append(skipPath) + self.session.bulk_save_objects(skipPathList) + self.session.commit() + return True + + @decorators.catch_sqlalchemy + def stopServer(self): + ''' + ''' + self.session.commit() + sys.exit(0) + + def __init__(self, session, lockDB): + self.session = session + self.report_ident = sqlalchemy.orm.query.Bundle('report_ident', + Report.id, + Report.bug_id, + Report.run_id, + Report.start_bugevent, + Report.start_bugpoint) + + +def run_server(port, db_uri, db_version_info, callback_event=None): + LOG.debug('Starting codechecker server ...') + + try: + engine = database_handler.SQLServer.create_engine(db_uri) + + LOG.debug('Creating new database session') + session = CreateSession(engine) + + except sqlalchemy.exc.SQLAlchemyError as alch_err: + LOG.error(str(alch_err)) + sys.exit(1) + + session.autoflush = False # Autoflush is enabled by default. + + LOG.debug('Starting thrift server') + try: + # Start thrift server. + handler = CheckerReportHandler(session, True) + + processor = CheckerReport.Processor(handler) + transport = TSocket.TServerSocket(port=port) + tfactory = TTransport.TBufferedTransportFactory() + pfactory = TBinaryProtocol.TBinaryProtocolFactory() + + server = TServer.TThreadPoolServer(processor, + transport, + tfactory, + pfactory, + daemon=True) + + LOG.info('Waiting for check results on ['+str(port)+']') + if callback_event: + callback_event.set() + LOG.debug('Starting to serve') + server.serve() + session.commit() + except socket.error as sockerr: + LOG.error(str(sockerr)) + if sockerr.errno == errno.EADDRINUSE: + LOG.error('Checker port '+str(port)+' is already used') + sys.exit(1) + except Exception as err: + LOG.error(str(err)) + session.commit() + sys.exit(1) Index: tools/codechecker/libcodechecker/suppress_file_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/suppress_file_handler.py @@ -0,0 +1,150 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +''' suppress file format + +# this is the old format +123324353456463442341242342343#1 || bug hash comment + +# this is the new format +123324353456463442341242342343#1 || filename || bug hash comment + +after removing the hash_value_type the generated format is: +123324353456463442341242342343 || filename || bug hash comment + +for backward compatibility the hash_value_type is an optional filed +''' + +import re +import os +import codecs +from libcodechecker import logger + +LOG = logger.get_new_logger('SUPPRESS_FILE_HANDLER') + + +COMMENT_SEPARATOR = '||' +HASH_TYPE_SEPARATOR = '#' + + +def get_suppress_data(suppress_file): + """ + processs a file object for suppress information + """ + + old_format_pattern = r"^(?P[\d\w]{32})(\#(?P\d))?\s*\|\|\s*(?P[^\|]*)$" + old_format = re.compile(old_format_pattern, re.UNICODE) + + new_format_pattern = r"^(?P[\d\w]{32})(\#(?P\d))?\s*\|\|\s*(?P[^\\\|]+)\s*\|\|\s*(?P[^\|]*)$" + new_format = re.compile(new_format_pattern, re.UNICODE) + + suppress_data = [] + + for line in suppress_file: + + new_format_match = re.match(new_format, line.strip()) + if new_format_match: + LOG.debug('Match for new suppress entry format:') + new_format_match = new_format_match.groupdict() + LOG.debug(new_format_match) + suppress_data.append((new_format_match['bug_hash'], + new_format_match['file_name'], + new_format_match['comment'])) + continue + + old_format_match = re.match(old_format, line.strip()) + if old_format_match: + LOG.debug('Match for old suppress entry format:') + old_format_match = old_format_match.groupdict() + LOG.debug(old_format_match) + suppress_data.append((old_format_match['bug_hash'], + u'', # empty file name + old_format_match['comment'])) + continue + + if line.strip() != '': + LOG.warning('Malformed suppress line: ' + line) + + return suppress_data + +# --------------------------------------------------------------------------- +def write_to_suppress_file(suppress_file, value, file_name, comment=''): + + comment = comment.decode('UTF-8') + + LOG.debug('Processing suppress file: '+suppress_file) + + try: + with codecs.open(suppress_file, 'r', 'UTF-8') as s_file: + suppress_data = get_suppress_data(s_file) + + if not os.stat(suppress_file)[6] == 0: + # file is not empty + + res = filter(lambda x: (x[0] == value and x[1] == file_name) or (x[0] == value and x[1] == ''), suppress_data) + + if res: + LOG.debug("Already found in\n %s" % (suppress_file)) + return True + + s_file = codecs.open(suppress_file, 'a', 'UTF-8') + + s_file.write(value+COMMENT_SEPARATOR+file_name+COMMENT_SEPARATOR+comment+'\n') + s_file.close() + + return True + + except Exception as ex: + LOG.error(str(ex)) + LOG.error("Failed to write: %s" % (suppress_file)) + return False + + +def remove_from_suppress_file(suppress_file, value, file_name): + """ + remove suppress information from the suppress file + old and new format is supported + """ + + LOG.debug('Removing ' + value + ' from \n' + suppress_file) + + try: + s_file = codecs.open(suppress_file, 'r+', 'UTF-8') + lines = s_file.readlines() + + # filter out new format first because it is more specific + old_format_pattern = r"^" + value + r"(\#\d)?\s*\|\|\s*(?P[^\|]*)$" + old_format = re.compile(old_format_pattern, re.UNICODE) + + new_format_pattern = r"^" + value + r"(\#d)?\s*\|\|\s*" + file_name + r"\s*\|\|\s*(?P[^\|]*)$" + new_format = re.compile(new_format_pattern, re.UNICODE) + + def check_for_match(line): + """ + check if the line matches the new or old format + """ + line = line.strip() + if re.match(new_format, line.strip()): + return False + if re.match(old_format, line.strip()): + return False + else: + return True + + # filter out lines which should be removed + lines = filter(lambda line: check_for_match(line), lines) + + s_file.seek(0) + s_file.truncate() + s_file.writelines(lines) + s_file.close() + + return True + + except Exception as ex: + LOG.error(str(ex)) + LOG.error("Failed to write: %s" % (suppress_file)) + return False Index: tools/codechecker/libcodechecker/suppress_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/suppress_handler.py @@ -0,0 +1,179 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +''' +suppress handling +''' + +import re +import abc +import linecache + +from libcodechecker import logger + +LOG = logger.get_new_logger('SUPPRESS HANDLER') + + +class SuppressHandler(object): + """ suppress handler base class """ + + __metaclass__ = abc.ABCMeta + + __suppressfile = None + + @abc.abstractmethod + def store_suppress_bug_id(self, + source_file_path, + bug_id, + file_name, + comment): + """ store the suppress bug_id """ + pass + + @abc.abstractmethod + def remove_suppress_bug_id(self, + bug_id, + file_name): + """ remove the suppress bug_id """ + pass + + @property + def suppress_file(self): + """" file on the filesystem where the suppress + data will be written """ + return self.__suppressfile + + @suppress_file.setter + def suppress_file(self, value): + """ set the suppress file""" + self.__suppressfile = value + + +class SourceSuppressHandler(object): + """ + handle bug suppression in the source + """ + + suppress_marker = 'codechecker_suppress' + + def __init__(self, source_file, bug_line): + """ + source line number indexing starts at 1 + """ + + self.__source_file = source_file + self.__bug_line = bug_line + self.__suppressed_checkers = [] + self.__suppress_comment = None + + def __check_if_comment(self, line): + """ + check if the line is a comment + accepted comment format is only if line starts with '//' + """ + return line.strip().startswith('//') + + def __process_suppress_info(self, source_section): + """ + return true if suppress comment found and matches the + required format + + Accepted source suppress format only above the bug line no + empty lines are accepted between the comment and the bug line + + For suppressing all checker results: + // codechecker_suppress [all] some multi line + // comment + + For suppressing some specific checker results: + // codechecker_suppress [checker.name1, checker.name2] some + // multi line comment + """ + nocomment = source_section.replace('//', '') + # remove extra spaces if any + formatted = ' '.join(nocomment.split()) + + # check for codechecker suppress comment + pattern = r'^\s*codechecker_suppress\s*\[\s*(?P(.*))\s*\]\s*(?P.*)$' + + ptn = re.compile(pattern) + + res = re.match(ptn, formatted) + + if res: + checkers = res.group('checkers') + if checkers == "all": + self.__suppressed_checkers.append('all') + else: + suppress_checker_list = re.findall(r"[^,\s]+", checkers.strip()) + self.__suppressed_checkers.extend(suppress_checker_list) + comment = res.group('comment') + if comment == '': + self.__suppress_comment = "WARNING! suppress comment is missing" + else: + self.__suppress_comment = res.group('comment') + return True + else: + return False + + def check_source_suppress(self): + """ + return true if there is a suppress comment or false if not + """ + + source_file = self.__source_file + LOG.debug('Checking for suppress comment in the source file: ' + + self.__source_file) + previous_line_num = self.__bug_line - 1 + suppression_result = False + if previous_line_num > 0: + + marker_found = False + comment_line = True + + collected_lines = [] + + while (not marker_found and comment_line): + source_line = linecache.getline(source_file, previous_line_num) + if(self.__check_if_comment(source_line)): + # it is a comment + if self.suppress_marker in source_line: + # found the marker + collected_lines.append(source_line.strip()) + marker_found = True + break + else: + collected_lines.append(source_line.strip()) + comment_line = True + else: + # this is not a comment + comment_line = False + break + + if(previous_line_num > 0): + previous_line_num -= 1 + else: + break + + # collected comment lines upward from bug line + rev = list(reversed(collected_lines)) + if marker_found: + suppression_result = self.__process_suppress_info(''.join(rev)) + + LOG.debug('Suppress comment found: ' + str(suppression_result)) + return suppression_result + + def suppressed_checkers(self): + """ + get the suppressed checkers list + """ + return self.__suppressed_checkers + + def suppress_comment(self): + """ + get the suppress comment + """ + return self.__suppress_comment Index: tools/codechecker/libcodechecker/tidy_output_converter.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/tidy_output_converter.py @@ -0,0 +1,391 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +This module is responsible for parsing clang-tidy output and generating plist +for the plist_parser module. +''' + +import re +import os +import copy +import plistlib + +from libcodechecker import logger + +LOG = logger.get_new_logger('TIDY_OUTPUT_HANDLER') + +class Note(object): + ''' + Represents a note and also this is the base class of Message. + ''' + + def __init__(self, path, line, column, message): + self.path = path + self.line = line + self.column = column + self.message = message + + + def __eq__(self, other): + return self.path == other.path and \ + self.line == other.line and \ + self.column == other.column and \ + self.message == other.message + + + def __str__(self): + return ('path=%s, line=%d, column=%s, message=%s') % \ + (self.path, self.line, self.column, self.message) + + +class Message(Note): + ''' + Represents a clang-tidy message with an optional fixit message. + ''' + + def __init__(self, path, line, column, message, checker, fixits=None, notes=None): + super(Message, self).__init__(path, line, column, message) + self.checker = checker + self.fixits = fixits if fixits else [] + self.notes = notes if notes else [] + + + def __eq__(self, other): + return super(Message, self).__eq__(other) and \ + self.checker == other.checker and \ + self.fixits == other.fixits and \ + self.notes == other.notes + + + def __str__(self): + return ('%s, checker=%s, fixits=%s, notes=%s') % \ + (super(Message, self).__str__(), self.checker, + [str(fixit) for fixit in self.fixits], + [str(note) for note in self.notes]) + + +class OutputParser(object): + ''' + Parser for clang-tidy console output. + ''' + + # Regex for parsing a clang-tidy message + message_line_re = re.compile( + # File path followed by a ':' + '^(?P[\S]+):' + # Line number followed by a ':' + '(?P\d+):' + # Column number followed by a ':' and a space + '(?P\d+):\ ' + # Severity followed by a ':' + '(?P\w+):' + # Checker message + '(?P[\S \t]+)\s*' + # Checker name + '\[(?P.*)\]') + + # Matches a note + note_line_re = re.compile( + # File path followed by a ':' + '^(?P[\S]+):' + # Line number followed by a ':' + '(?P\d+):' + # Column number followed by a ':' and a space + '(?P\d+):\ ' + # Severity == note + 'note:' + # Checker message + '(?P.*)') + + def __init__(self): + self.messages = [] + + + def parse_messages_from_file(self, path): + ''' + Parse clang-tidy output dump (redirected output). + ''' + + with open(path, 'r') as file: + return self.parse_messages(file) + + def parse_messages(self, tidy_out): + ''' + Parse the given clang-tidy output. This method calls iter(tidy_out). + The iterator should return lines. + + Parameters: + tidy_out: something iterable (e.g.: a file object) + ''' + + titer = iter(tidy_out) + try: + next_line = titer.next() + while True: + message, next_line = self._parse_message(titer, next_line) + if message != None: + self.messages.append(message) + except StopIteration: + pass + + return self.messages + + def _parse_message(self, titer, line): + ''' + Parse the given line. Returns a (message, next_line) pair or throws a + StopIteration. The message could be None. + + Parameters: + titer: clang-tidy output iterator + line: the current line + ''' + + match = OutputParser.message_line_re.match(line) + if match is None: + return None, titer.next() + + message = Message( + os.path.abspath(match.group('path')), + int(match.group('line')), + int(match.group('column')), + match.group('message').strip(), + match.group('checker').strip()) + + try: + line = titer.next() + line = self._parse_code(message, titer, line) + line = self._parse_fixits(message, titer, line) + line = self._parse_notes(message, titer, line) + + return message, line + except StopIteration: + return message, line + + + def _parse_code(self, message, titer, line): + # eat code line + if OutputParser.note_line_re.match(line) or \ + OutputParser.message_line_re.match(line): + LOG.debug("Unexpected line: %s. Expected a code line!" % line) + return line + + # eat arrow line + # FIXME: range support? + line = titer.next() + if '^' not in line: + LOG.debug("Unexpected line: %s. Expected an arrow line!" % line) + return line + return titer.next() + + + def _parse_fixits(self, message, titer, line): + '''Parses fixit messages''' + + while OutputParser.message_line_re.match(line) is None and \ + OutputParser.note_line_re.match(line) is None: + message_text = line.strip() + if message_text == '': + continue + + message.fixits.append(Note(message.path, message.line, + line.find(message_text) + 1, message_text)) + line = titer.next() + return line + + + def _parse_notes(self, message, titer, line): + '''Parses note messages''' + + while OutputParser.message_line_re.match(line) is None: + match = OutputParser.note_line_re.match(line) + if match is None: + LOG.debug("Unexpected line: %s" % line) + return titer.next() + + message.notes.append(Note(os.path.abspath(match.group('path')), + int(match.group('line')), + int(match.group('column')), + match.group('message').strip())) + line = titer.next() + line = self._parse_code(message, titer, line) + return line + + +class PListConverter(object): + ''' + clang-tidy messages to plist converter. + ''' + + def __init__(self): + self.plist = { + 'files' : [], + 'diagnostics' : [] + } + + + def _add_files_from_messages(self, messages): + ''' + Adds the new files from the given message array to the plist's "files" + key, and returns a path to file index dictionary. + ''' + + fmap = {} + for message in messages: + try: + # This file is already in the plist + idx = self.plist['files'].index(message.path) + fmap[message.path] = idx + except ValueError: + # New file + fmap[message.path] = len(self.plist['files']) + self.plist['files'].append(message.path) + + # collect file paths from the message notes + for nt in message.notes: + try: + # This file is already in the plist + idx = self.plist['files'].index(nt.path) + fmap[nt.path] = idx + except ValueError: + # New file + fmap[nt.path] = len(self.plist['files']) + self.plist['files'].append(nt.path) + + return fmap + + + def _add_diagnostics(self, messages, fmap): + ''' + Adds the messages to the plist as diagnostics. + ''' + + for message in messages: + diag = PListConverter._create_diag(message, fmap) + self.plist['diagnostics'].append(diag) + + + @staticmethod + def _get_checker_category(checker): + ''' + Returns the check's category. + ''' + + parts = checker.split('-') + if len(parts) == 0: + # I don't know if it's possible + return 'unknown' + else: + return parts[0] + + + @staticmethod + def _create_diag(message, fmap): + ''' + Creates a new plist diagnostic from a single clang-tidy message. + ''' + + diag = {} + diag['location'] = PListConverter._create_location(message, fmap) + diag['check_name'] = message.checker + diag['description'] = message.message + diag['category'] = PListConverter._get_checker_category(message.checker) + diag['type'] = 'clang-tidy' + diag['path'] = [PListConverter._create_event_from_note(message, fmap)] + + PListConverter._add_fixits(diag, message, fmap) + PListConverter._add_notes(diag, message, fmap) + + return diag + + + @staticmethod + def _create_location(note, fmap): + return { + 'line' : note.line, + 'col' : note.column, + 'file' : fmap[note.path] + } + + + @staticmethod + def _create_event_from_note(note, fmap): + return { + 'kind' : 'event', + 'location' : PListConverter._create_location(note, fmap), + 'depth' : 0, # I don't know WTF is this + 'message' : note.message + } + + + @staticmethod + def _create_edge(start_note, end_note, fmap): + start_loc = PListConverter._create_location(start_note, fmap) + end_loc = PListConverter._create_location(end_note, fmap) + return { + 'start' : [start_loc, start_loc], + 'end' : [end_loc, end_loc] + } + + + @staticmethod + def _add_fixits(diag, message, fmap): + ''' + Adds fixits as events to the diagnostics. + ''' + + for fixit in message.fixits: + mf = copy.deepcopy(fixit) + mf.message = '%s (fixit)' % fixit.message + diag['path'].append(PListConverter._create_event_from_note( + mf, fmap)) + + + @staticmethod + def _add_notes(diag, message, fmap): + ''' + Adds notes as events to the diagnostics. It also creates edges between + the notes. + ''' + + edges = [] + last = None + for note in message.notes: + if last is not None: + edges.append(PListConverter._create_edge(last, note, fmap)) + diag['path'].append(PListConverter._create_event_from_note( + note, fmap)) + last = note + + diag['path'].append({ + 'kind' : 'control', + 'edges' : edges + }) + + + def add_messages(self, messages): + ''' + Adds the given clang-tidy messages to the plist. + ''' + + fmap = self._add_files_from_messages(messages) + self._add_diagnostics(messages, fmap) + + + def write_to_file(self, path): + ''' + Writes out the plist XML to the given path. + ''' + + with open(path, 'wb') as file: + self.write(file) + + + def write(self, file): + ''' + Writes out the plist XML using the given file object. + ''' + + plistlib.writePlist(self.plist, file) Index: tools/codechecker/libcodechecker/util.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/util.py @@ -0,0 +1,198 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +util module +''' + +import datetime +import os +import hashlib +import ntpath +import sys +import glob +import socket +import shutil +import subprocess + +from libcodechecker import logger + +# WARNING! LOG should be only used in this module +LOG = logger.get_new_logger('UTIL') + + +# --------------------------------------------------------------------- +def get_free_port(): + ''' get a free port from the os''' + + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.bind(('', 0)) + free_port = s.getsockname()[1] + s.close() + + return free_port + + +# --------------------------------------------------------------------- +def is_localhost(address): + ''' + Check if address is one of the valid values and try to get the + IP-addresses from the system. + ''' + + valid_values = ['localhost', '0.0.0.0', '*'] + + try: + valid_values.append(socket.gethostbyname('localhost')) + except Exception: + # failed to get ip address for localhost + pass + + try: + valid_values.append(socket.gethostbyname(socket.gethostname())) + except Exception: + # failed to get ip address for host_name + pass + + return address in valid_values + + +# --------------------------------------------------------------------- +def match_file_name(file_name, pattern): + file_name_parts = file_name.split('--') + + if file_name_parts[0] == pattern: + # print('%s == %s : TRUE'% (file_name_parts[0],pattern)) + return True + else: + # print('%s == %s : TRUE'% (file_name_parts[0],pattern)) + return False + + +# --------------------------------------------------------------------- +def get_file_last_modification_time(file): + ''' + Returns the last modification time of a file + ''' + return datetime.datetime.fromtimestamp(os.path.getmtime(file)) + + +# --------------------------------------------------------------------- +def get_env_var(env_var, needed=False): + ''' + Read the environment variables and handle the exception if a necessary + environment variable is missing + ''' + + value = os.getenv(env_var) + if needed and not value: + LOG.critical('Failed to read necessary environment variable %s.' + ' (Maybe codechecker was not configured properly)' + % (env_var)) + sys.exit(1) + + return value + + +# ------------------------------------------------------------------------- +def get_tmp_dir_hash(): + '''Generate a hash based on the current time and process id.''' + + pid = os.getpid() + time = datetime.datetime.now() + + data = str(pid) + str(time) + + dir_hash = hashlib.md5() + dir_hash.update(data) + + LOG.debug('The generated temporary directory hash is %s' + % (dir_hash.hexdigest())) + + return dir_hash.hexdigest() + + +# ------------------------------------------------------------------------- +def get_file_name_from_path(path): + '''Get the filename from a path.''' + head, tail = ntpath.split(path) + return head, tail + + +# ------------------------------------------------------------------------- +def get_obj_target(object_file_path): + return os.path.split(os.path.abspath(dir))[-2] + + +# ------------------------------------------------------------------------- +def create_dir(path): + '''Create a directory safely if it does not exist yet. + This may be called from several processes or threads, creating the same + directory, and it fails only if the directory is not created. + ''' + + if not os.path.isdir(path): + try: + LOG.debug('Creating directory %s' % (path)) + os.makedirs(path) + except Exception as e: + if not os.path.isdir(path): + LOG.error('Failed to create directory %s' % (path)) + raise e + + return + + +# ------------------------------------------------------------------------- +def get_file_list(path, pattern): + glob_pattern = os.path.join(path, pattern) + return glob.glob(glob_pattern) + + +# ------------------------------------------------------------------------- +def remove_file_list(file_list): + for rfile in file_list: + LOG.debug(rfile) + try: + os.remove(rfile) + except OSError: + # maybe another thread has already deleted it + LOG.debug('Failed to remove file %s' % (rfile)) + + return + + +# ------------------------------------------------------------------------- +def remove_dir(path): + + def error_handler(*args): + LOG.warning('Failed to remove directory %s' % (path)) + + shutil.rmtree(path, onerror=error_handler) + + +def call_command(command, env=None): + ''' Call an external command and return with (output, return_code).''' + + try: + LOG.debug('Run ' + ' '.join(command)) + out = subprocess.check_output(command, + bufsize=-1, + env=env, + stderr=subprocess.STDOUT) + LOG.debug(out) + return out, 0 + except subprocess.CalledProcessError as ex: + LOG.debug('Running command "' + ' '.join(command) + '" Failed') + LOG.debug(str(ex.returncode)) + LOG.debug(ex.output) + return ex.output, ex.returncode + +def get_default_workspace(): + """ + default workspace in the users home directory + """ + workspace = os.path.join(os.path.expanduser("~"), '.codechecker') + return workspace Index: tools/codechecker/libcodechecker/viewer_server/__init__.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/viewer_server/__init__.py @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. Index: tools/codechecker/libcodechecker/viewer_server/client_db_access_handler.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/viewer_server/client_db_access_handler.py @@ -0,0 +1,1295 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +Handle thrift requests +''' +import zlib +import os +import datetime +from collections import defaultdict +import ntpath +import codecs + +import sqlalchemy +from sqlalchemy import asc, desc +from sqlalchemy.sql import or_, and_, func +from sqlalchemy.sql.expression import literal_column + +from libcodechecker.db_model.orm_model import * + +import shared +from codeCheckerDBAccess import constants +from codeCheckerDBAccess.ttypes import * + +from libcodechecker import logger + +LOG = logger.get_new_logger('ACCESS HANDLER') + + +# ----------------------------------------------------------------------- +def timefunc(function): + ''' + timer function + ''' + + func_name = function.__name__ + + def debug_wrapper(*args, **kwargs): + ''' + wrapper for debug log + ''' + before = datetime.now() + res = function(*args, **kwargs) + after = datetime.now() + timediff = after - before + diff = timediff.microseconds/1000 + LOG.debug('['+str(diff)+'ms] ' + func_name) + return res + + def release_wrapper(*args, **kwargs): + ''' + no logging + ''' + res = function(*args, **kwargs) + return res + + if logger.get_log_level() == logger.DEBUG: + return debug_wrapper + else: + return release_wrapper + + +def conv(text): + ''' + Convert * to % got from clients for the database queries + ''' + if text is None: + return '%' + return text.replace('*', '%') + + +def construct_report_filter(report_filters): + ''' + construct the report filter for reports and suppressed reports + ''' + + OR = [] + if report_filters is None: + AND = [] + AND.append(Report.checker_message.like('%')) + AND.append(Report.checker_id.like('%')) + AND.append(File.filepath.like('%')) + + OR.append(and_(*AND)) + filter_expression = or_(*OR) + return filter_expression + + for report_filter in report_filters: + AND = [] + if report_filter.checkerMsg: + AND.append(Report.checker_message.ilike( + conv(report_filter.checkerMsg))) + if report_filter.checkerId: + AND.append(Report.checker_id.ilike( + conv(report_filter.checkerId))) + if report_filter.filepath: + AND.append(File.filepath.ilike( + conv(report_filter.filepath))) + if report_filter.severity is not None: + # severity value can be 0 + AND.append(Report.severity == report_filter.severity) + if report_filter.suppressed: + AND.append(Report.suppressed == True) + else: + AND.append(Report.suppressed == False) + + OR.append(and_(*AND)) + + filter_expression = or_(*OR) + + return filter_expression + + +class ThriftRequestHandler(): + ''' + Connect to database and handle thrift client requests + ''' + + def __init__(self, + session, + checker_md_docs, + checker_md_docs_map, + suppress_handler, + db_version_info): + + self.__checker_md_docs = checker_md_docs + self.__checker_doc_map = checker_md_docs_map + self.__suppress_handler = suppress_handler + self.__session = session + + def __queryReport(self, reportId): + session = self.__session + + try: + q = session.query(Report, + File, + BugPathEvent, + SuppressBug) \ + .filter(Report.id == reportId) \ + .outerjoin(File, + Report.file_id == File.id) \ + .outerjoin(BugPathEvent, + Report.end_bugevent == BugPathEvent.id) \ + .outerjoin(SuppressBug, + SuppressBug.hash == Report.bug_id) + + results = q.limit(1).all() + if len(results) < 1: + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.DATABASE, + "Report " + reportId + " not found!") + + report, source_file, lbpe, suppress_bug = results[0] + + last_event_pos = \ + shared.ttypes.BugPathEvent( + startLine=lbpe.line_begin, + startCol=lbpe.col_begin, + endLine=lbpe.line_end, + endCol=lbpe.col_end, + msg=lbpe.msg, + fileId=lbpe.file_id, + filePath=source_file.filepath) + + if suppress_bug: + suppress_comment = suppress_bug.comment + else: + suppress_comment = None + + return ReportData( + bugHash=report.bug_id, + checkedFile=source_file.filepath, + checkerMsg=report.checker_message, + suppressed=report.suppressed, + reportId=report.id, + fileId=source_file.id, + lastBugPosition=last_event_pos, + checkerId=report.checker_id, + severity=report.severity, + suppressComment=suppress_comment) + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.DATABASE, + msg) + + + def __sortResultsQuery(self, query, sort_types=None): + ''' + Helper method for __queryDiffResults and __queryResults to apply sorting. + ''' + + # get a list of sort_types which will be a nested ORDER BY + sort_type_map = {} + sort_type_map[SortType.FILENAME] = [File.filepath, BugPathEvent.line_begin] + sort_type_map[SortType.CHECKER_NAME] = [Report.checker_id] + sort_type_map[SortType.SEVERITY] = [Report.severity] + + # mapping the SQLAlchemy functions + order_type_map = {} + order_type_map[Order.ASC] = asc + order_type_map[Order.DESC] = desc + + if sort_types is None: + sort_types = [SortMode(SortType.FILENAME, Order.ASC)] + + for sort in sort_types: + sorttypes = sort_type_map.get(sort.type) + for sorttype in sorttypes: + order_type = order_type_map.get(sort.ord) + query = query.order_by(order_type(sorttype)) + + return query + + + def __queryResults(self, run_id, limit, offset, sort_types, report_filters): + + max_query_limit = constants.MAX_QUERY_SIZE + if limit > max_query_limit: + LOG.debug('Query limit ' + str(limit) + + ' was larger than max query limit ' + + str(max_query_limit) + ', setting limit to ' + + str(max_query_limit)) + limit = max_query_limit + + session = self.__session + filter_expression = construct_report_filter(report_filters) + + try: + + q = session.query(Report, + File, + BugPathEvent, + SuppressBug) \ + .filter(Report.run_id == run_id) \ + .outerjoin(File, + and_(Report.file_id == File.id, + File.run_id == run_id)) \ + .outerjoin(BugPathEvent, + Report.end_bugevent == BugPathEvent.id) \ + .outerjoin(SuppressBug, + and_(SuppressBug.hash == Report.bug_id, + SuppressBug.run_id == run_id)) \ + .filter(filter_expression) + + q = self.__sortResultsQuery(q, sort_types) + + results = [] + for report, source_file, lbpe, suppress_bug in \ + q.limit(limit).offset(offset): + + last_event_pos = \ + shared.ttypes.BugPathEvent(startLine=lbpe.line_begin, + startCol=lbpe.col_begin, + endLine=lbpe.line_end, + endCol=lbpe.col_end, + msg=lbpe.msg, + fileId=lbpe.file_id, + filePath=source_file.filepath) + + if suppress_bug: + suppress_comment = suppress_bug.comment + else: + suppress_comment = None + + results.append( + ReportData(bugHash=report.bug_id, + checkedFile=source_file.filepath, + checkerMsg=report.checker_message, + suppressed=report.suppressed, + reportId=report.id, + fileId=source_file.id, + lastBugPosition=last_event_pos, + checkerId=report.checker_id, + severity=report.severity, + suppressComment=suppress_comment) + ) + + return results + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def getRunData(self): + + session = self.__session + results = [] + try: + # count the reports subquery + stmt = session.query(Report.run_id, + func.count(literal_column('*')).label('report_count')) \ + .filter(Report.suppressed == False) \ + .group_by(Report.run_id) \ + .subquery() + + q = session.query(Run, stmt.c.report_count) \ + .outerjoin(stmt, Run.id == stmt.c.run_id) \ + .order_by(Run.date) + + for instance, reportCount in q: + if reportCount is None: + reportCount = 0 + + results.append(RunData(instance.id, + str(instance.date), + instance.name, + instance.duration, + reportCount, + instance.command + )) + return results + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def getReport(self, reportId): + return self.__queryReport(reportId) + + @timefunc + def getRunResults(self, run_id, limit, offset, sort_types, report_filters): + + return self.__queryResults(run_id, + limit, + offset, + sort_types, + report_filters) + + @timefunc + def getRunResultCount(self, run_id, report_filters): + + filter_expression = construct_report_filter(report_filters) + + session = self.__session + try: + reportCount = session.query(Report) \ + .filter(Report.run_id == run_id) \ + .outerjoin(File, + and_(Report.file_id == File.id, + File.run_id == run_id)) \ + .outerjoin(BugPathEvent, + Report.end_bugevent == BugPathEvent.id) \ + .outerjoin(SuppressBug, + and_(SuppressBug.hash == Report.bug_id, + SuppressBug.run_id == run_id)) \ + .filter(filter_expression)\ + .count() + + if reportCount is None: + reportCount = 0 + + return reportCount + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def __construct_bug_event_list(self, session, start_bug_event): + + file_path_cache = {} + bug_events = [] + event = session.query(BugPathEvent).get(start_bug_event) + + file_path = file_path_cache.get(event.file_id) + if not file_path: + f = session.query(File).get(event.file_id) + file_path = f.filepath + file_path_cache[event.file_id] = file_path + + bug_events.append((event, file_path)) + + while event.next is not None: + + event = session.query(BugPathEvent).get(event.next) + + file_path = file_path_cache.get(event.file_id) + if not file_path: + f = session.query(File).get(event.file_id) + file_path = f.filepath + file_path_cache[event.file_id] = file_path + + bug_events.append((event, file_path)) + + return bug_events + + @timefunc + def __construct_bug_point_list(self, session, start_bug_point): + # start_bug_point can be None + + file_path_cache = {} + bug_points = [] + + if start_bug_point: + + bug_point = session.query(BugReportPoint).get(start_bug_point) + file_path = file_path_cache.get(bug_point.file_id) + if not file_path: + f = session.query(File).get(bug_point.file_id) + file_path = f.filepath + file_path_cache[bug_point.file_id] = file_path + + bug_points.append((bug_point, file_path)) + while bug_point.next is not None: + + bug_point = session.query(BugReportPoint).get(bug_point.next) + + file_path = file_path_cache.get(bug_point.file_id) + if not file_path: + f = session.query(File).get(bug_point.file_id) + file_path = f.filepath + file_path_cache[bug_point.file_id] = file_path + + bug_points.append((bug_point, file_path)) + + return bug_points + + @timefunc + def getReportDetails(self, reportId): + """ + Parameters: + - reportId + """ + + session = self.__session + try: + report = session.query(Report).get(reportId) + + events = self.__construct_bug_event_list(session, + report.start_bugevent) + bug_events_list = [] + for (event, file_path) in events: + bug_events_list.append( + shared.ttypes.BugPathEvent( + startLine=event.line_begin, + startCol=event.col_begin, + endLine=event.line_end, + endCol=event.col_end, + msg=event.msg, + fileId=event.file_id, + filePath=file_path)) + + points = self.__construct_bug_point_list(session, + report.start_bugpoint) + + bug_point_list = [] + for (bug_point, file_path) in points: + bug_point_list.append( + shared.ttypes.BugPathPos( + startLine=bug_point.line_begin, + startCol=bug_point.col_begin, + endLine=bug_point.line_end, + endCol=bug_point.col_end, + fileId=bug_point.file_id, + filePath=file_path)) + + return ReportDetails(bug_events_list, bug_point_list) + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + def __set_report_suppress_flag(self, + session, + run_ids, + bug_id_hash, + source_file_name, + suppress_flag): + """ + update the suppress flag for multiple report entries based on the + filter + """ + + if not run_ids: + # there are no run ids where the report should be suppressed + return + + def check_filename(data): + report, file_obj = data + source_file_path, f_name = ntpath.split(file_obj.filepath) + if f_name == source_file_name: + return True + else: + return False + + reports = session.query(Report, File) \ + .filter(and_(Report.bug_id == bug_id_hash, + Report.run_id.in_(run_ids))) \ + .outerjoin(File, File.id == Report.file_id) \ + .all() + + reports = filter(check_filename, reports) + + for report, file_obj in reports: + report.suppressed = suppress_flag + + def __update_suppress_storage_data(self, + run_ids, + report, + suppress, + comment=u''): + """ + update suppress information in the database and in the suppress file + can be used to suppress or unsuppress a report for multiple runs + """ + session = self.__session + + report_id = report.id + bug_id_hash = report.bug_id + + source_file = session.query(File).get(report.file_id) + source_file_path, source_file_name = ntpath.split(source_file.filepath) + + LOG.debug('Updating suppress data for: ' + str(report_id) + ' bug id ' + + bug_id_hash + ' file name ' + source_file_name + + ' suppressing ' + str(suppress)) + + # check if it is already suppressed for any run ids + suppressed = session.query(SuppressBug) \ + .filter(or_( \ + and_(SuppressBug.hash == bug_id_hash, + SuppressBug.file_name == source_file_name, + SuppressBug.run_id.in_(run_ids)), + and_(SuppressBug.hash == bug_id_hash, + SuppressBug.file_name == '', + SuppressBug.run_id.in_(run_ids)) + )) \ + .all() + + if not suppressed and suppress: + # the bug is not suppressed for any run_id, suppressing it + LOG.debug('Bug is not suppressed in any runs') + for rId in run_ids: + suppress_bug = SuppressBug(rId, + bug_id_hash, + source_file_name, + comment) + session.add(suppress_bug) + + # update report entries + self.__set_report_suppress_flag(session, + run_ids, + bug_id_hash, + source_file_name, + suppress_flag=suppress) + + elif suppressed and suppress: + # already suppressed for some run ids check if other suppression + # is needed for other run id + suppressed_runids = set([r.run_id for r in suppressed]) + LOG.debug('Bug is suppressed in these runs:' + + ' '.join([str(r) for r in suppressed_runids])) + suppress_in_these_runs = set(run_ids).difference(suppressed_runids) + for run_id in suppress_in_these_runs: + suppress_bug = SuppressBug(run_id, + bug_id_hash, + source_file_name, + comment) + session.add(suppress_bug) + self.__set_report_suppress_flag(session, + suppress_in_these_runs, + bug_id_hash, + source_file_name, + suppress_flag=suppress) + + elif suppressed and not suppress: + # already suppressed for some run ids + # remove those entries + already_suppressed_runids = \ + filter(lambda bug: bug.run_id in run_ids, set(suppressed)) + + unsuppress_in_these_runs = \ + {bug.run_id for bug in already_suppressed_runids} + + LOG.debug('Already suppressed, unsuppressing now') + suppressed = session.query(SuppressBug) \ + .filter(and_(SuppressBug.hash == bug_id_hash, + SuppressBug.file_name == source_file_name, + SuppressBug.run_id.in_(unsuppress_in_these_runs))) + # delete supppress bug entries + for sp in suppressed: + session.delete(sp) + + # update report entries + self.__set_report_suppress_flag(session, + unsuppress_in_these_runs, + bug_id_hash, + source_file_name, + suppress_flag=suppress) + + #elif suppressed is None and not suppress: + # # check only in the file if there is anything that should be + # # removed the database has no entries in the suppressBug table + + ret = True + if suppress: + # store to suppress file + ret = self.__suppress_handler \ + .store_suppress_bug_id(bug_id_hash, + source_file_name, + comment) + else: + # remove from suppress file + ret = self.__suppress_handler \ + .remove_suppress_bug_id(bug_id_hash, + source_file_name) + + if not ret: + session.rollback() + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.IOERROR, + 'Failed to store suppress bug id') + else: + session.commit() + return True + + @timefunc + def suppressBug(self, run_ids, report_id, comment): + """ + Add suppress bug entry to the SuppressBug table. + Set the suppressed flag for the selected report. + """ + session = self.__session + try: + report = session.query(Report).get(report_id) + if report: + return self.__update_suppress_storage_data(run_ids, + report, + True, + comment) + else: + msg = 'Report id ' + str(report_id) + \ + ' was not found in the database.' + LOG.error(msg) + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.DATABASE, msg) + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.DATABASE, msg) + + except Exception as ex: + msg = str(ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.IOERROR, msg) + + @timefunc + def unSuppressBug(self, run_ids, report_id): + """ + Remove the suppress flag from the reports in multiple runs if given. + Cleanup the SuppressBug table to remove suppress entries. + """ + session = self.__session + try: + report = session.query(Report).get(report_id) + if report: + return self.__update_suppress_storage_data(run_ids, + report, + False) + else: + msg = 'Report id ' + str(report_id) + \ + ' was not found in the database.' + LOG.error(msg) + raise shared.ttypes.RequestFailed( + shared.ttypes.ErrorCode.DATABASE, msg) + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, + msg) + + except Exception as ex: + msg = str(ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.IOERROR, + msg) + + def getCheckerDoc(self, checkerId): + """ + Parameters: + - checkerId + """ + + text = "No documentation found for checker: " + checkerId + \ + "\n\nPlease refer to the documentation at the " + if "." in checkerId: + text += "[ClangSA](http://clang-analyzer.llvm.org/available_checks.html)" + elif "-" in checkerId: + text += "[ClangTidy](http://clang.llvm.org/extra/clang-tidy/checks/list.html)" + text += " homepage." + + try: + md_file = self.__checker_doc_map.get(checkerId) + if md_file: + md_file = os.path.join(self.__checker_md_docs, md_file) + with open(md_file, 'r') as md_content: + text = md_content.read() + + return text + + except Exception as ex: + msg = str(ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.IOERROR, msg) + + def getCheckerConfigs(self, run_id): + """ + Parameters: + - run_id + """ + session = self.__session + try: + configs = session.query(Config) \ + .filter(Config.run_id == run_id) \ + .all() + + configs = [(c.checker_name, c.attribute, c.value) + for c in configs] + res = [] + for cName, attribute, value in configs: + res.append(shared.ttypes.ConfigValue(cName, attribute, value)) + + return res + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def getSkipPaths(self, run_id): + session = self.__session + try: + suppressed_paths = session.query(SkipPath) \ + .filter(SkipPath.run_id == run_id) \ + .all() + + results = [] + for sp in suppressed_paths: + encoded_path = sp.path + encoded_comment = sp.comment + results.append(SkipPathData(encoded_path, encoded_comment)) + + return results + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def getBuildActions(self, reportId): + + session = self.__session + try: + build_actions = session.query(BuildAction) \ + .outerjoin(ReportsToBuildActions) \ + .filter(ReportsToBuildActions.report_id == reportId) \ + .all() + + return [ba.build_cmd for ba in build_actions] + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def getFileId(self, run_id, path): + + session = self.__session + + try: + sourcefile = session.query(File) \ + .filter(File.run_id == run_id, + File.filepath == path) \ + .first() + + if sourcefile is None: + return -1 + + return sourcefile.id + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def getSourceFileData(self, fileId, fileContent): + """ + Parameters: + - fileId + - fileContent + """ + session = self.__session + try: + sourcefile = session.query(File) \ + .filter(File.id == fileId).first() + + if sourcefile is None: + return SourceFileData() + + if fileContent and sourcefile.content: + source = zlib.decompress(sourcefile.content) + + source = codecs.decode(source, 'utf-8', 'replace') + + return SourceFileData(fileId=sourcefile.id, + filePath=sourcefile.filepath, + fileContent=source) + else: + return SourceFileData(fileId=sourcefile.id, + filePath=sourcefile.filepath) + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + @timefunc + def getRunResultTypes(self, run_id, report_filters): + + session = self.__session + try: + + filter_expression = construct_report_filter(report_filters) + + q = session.query(Report) \ + .filter(Report.run_id == run_id) \ + .outerjoin(File, + Report.file_id == File.id) \ + .outerjoin(BugPathEvent, + Report.end_bugevent == BugPathEvent.id) \ + .outerjoin(SuppressBug, + and_(SuppressBug.hash == Report.bug_id, + SuppressBug.run_id == run_id)) \ + .order_by(Report.checker_id) \ + .filter(filter_expression) \ + .all() + + count_results = defaultdict(int) + + result_reports = defaultdict() + # count and filter out the results for the same checker_id + for r in q: + count_results[r.checker_id] += 1 + result_reports[r.checker_id] = r + + results = [] + for checker_id, res in result_reports.items(): + + results.append(ReportDataTypeCount(res.checker_id, res.severity, count_results[res.checker_id])) + + # result count ascending + results = sorted(results, key=lambda rep: rep.count, reverse=True) + + return results + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + # ----------------------------------------------------------------------- + @timefunc + def __get_hashes_for_diff(self, session, base_run_id, new_run_id): + + LOG.debug('query all baseline hashes') + # keyed tuple list is returned + base_line_hashes = session.query(Report.bug_id) \ + .filter(Report.run_id == base_run_id) \ + .all() + # LOG.debug(len(base_line_hashes)) + + LOG.debug('query all new check hashes') + # keyed tuple list is returned + new_check_hashes = session.query(Report.bug_id) \ + .filter(Report.run_id == new_run_id) \ + .all() + # LOG.debug(len(new_check_hashes)) + + base_line_hashes = set([t[0] for t in base_line_hashes]) + new_check_hashes = set([t[0] for t in new_check_hashes]) + + return base_line_hashes, new_check_hashes + + + # ----------------------------------------------------------------------- + @timefunc + def __queryDiffResults(self, + session, + diff_hash_list, + run_id, + limit, + offset, + sort_types=None, + report_filters=None): + + max_query_limit = constants.MAX_QUERY_SIZE + if limit > max_query_limit: + LOG.debug('Query limit ' + str(limit) + + ' was larger than max query limit ' + + str(max_query_limit) + ', setting limit to ' + + str(max_query_limit)) + limit = max_query_limit + + filter_expression = construct_report_filter(report_filters) + + try: + q = session.query(Report, + File, + BugPathEvent, + SuppressBug) \ + .filter(Report.run_id == run_id) \ + .outerjoin(File, + and_(Report.file_id == File.id, + File.run_id == run_id)) \ + .outerjoin(BugPathEvent, + Report.end_bugevent == BugPathEvent.id) \ + .outerjoin(SuppressBug, + and_(SuppressBug.hash == Report.bug_id, + SuppressBug.run_id == run_id)) \ + .filter(Report.bug_id.in_(diff_hash_list)) \ + .filter(filter_expression) + + q = self.__sortResultsQuery(q, sort_types) + + results = [] + for report, source_file, lbpe, suppress_bug \ + in q.limit(limit).offset(offset): + + lastEventPos = \ + shared.ttypes.BugPathEvent(startLine=lbpe.line_begin, + startCol=lbpe.col_begin, + endLine=lbpe.line_end, + endCol=lbpe.col_end, + msg=lbpe.msg, + fileId=lbpe.file_id) + + if suppress_bug: + suppress_comment = suppress_bug.comment + else: + suppress_comment = None + results.append(ReportData(bugHash=report.bug_id, + checkedFile=source_file.filepath, + checkerMsg=report.checker_message, + suppressed=report.suppressed, + reportId=report.id, + fileId=source_file.id, + lastBugPosition=lastEventPos, + checkerId=report.checker_id, + severity=report.severity, + suppressComment=suppress_comment)) + + return results + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, msg) + + # ----------------------------------------------------------------------- + @timefunc + def getNewResults(self, + base_run_id, + new_run_id, + limit, + offset, + sort_types=None, + report_filters=None): + + session = self.__session + + base_line_hashes, new_check_hashes = \ + self.__get_hashes_for_diff(session, + base_run_id, + new_run_id) + + diff_hashes = list(new_check_hashes.difference(base_line_hashes)) + + LOG.debug(len(diff_hashes)) + LOG.debug(diff_hashes) + + if len(diff_hashes) == 0: + return [] + + return self.__queryDiffResults(session, + diff_hashes, + new_run_id, + limit, + offset, + sort_types, + report_filters) + + # ----------------------------------------------------------------------- + @timefunc + def getResolvedResults(self, + base_run_id, + new_run_id, + limit, + offset, + sort_types=None, + report_filters=None): + + session = self.__session + base_line_hashes, new_check_hashes = \ + self.__get_hashes_for_diff(session, + base_run_id, + new_run_id) + + diff_hashes = list(base_line_hashes.difference(new_check_hashes)) + + LOG.debug(len(diff_hashes)) + LOG.debug(diff_hashes) + + if len(diff_hashes) == 0: + return [] + + return self.__queryDiffResults(session, + diff_hashes, + base_run_id, + limit, + offset, + sort_types, + report_filters) + + # ----------------------------------------------------------------------- + @timefunc + def getUnresolvedResults(self, + base_run_id, + new_run_id, + limit, + offset, + sort_types=None, + report_filters=None): + + session = self.__session + base_line_hashes, new_check_hashes = \ + self.__get_hashes_for_diff(session, + base_run_id, + new_run_id) + + diff_hashes = list(base_line_hashes.intersection(new_check_hashes)) + + LOG.debug('diff hashes' + str(len(diff_hashes))) + LOG.debug(diff_hashes) + + if len(diff_hashes) == 0: + return [] + + return self.__queryDiffResults(session, + diff_hashes, + new_run_id, + limit, + offset, + sort_types, + report_filters) + + # ----------------------------------------------------------------------- + @timefunc + def getAPIVersion(self): + # returns the thrift api version + return constants.API_VERSION + + # ----------------------------------------------------------------------- + @timefunc + def removeRunResults(self, run_ids): + + session = self.__session + + runs_to_delete = [] + for run_id in run_ids: + LOG.debug('run ids to delete') + LOG.debug(run_id) + + run_to_delete = session.query(Run).get(run_id) + if not run_to_delete.can_delete: + LOG.debug("Can't delete " + str(run_id)) + continue + + run_to_delete.can_delete = False + session.commit() + runs_to_delete.append(run_to_delete) + + for run_to_delete in runs_to_delete: + session.delete(run_to_delete) + session.commit() + + return True + + # ----------------------------------------------------------------------- + def getSuppressFile(self): + """ + return the suppress file path or empty string if not set + """ + suppress_file = self.__suppress_handler.suppress_file + if suppress_file: + return suppress_file + return '' + + # ----------------------------------------------------------------------- + def __queryDiffResultsCount(self, + session, + diff_hash_list, + run_id, + report_filters=None): + """ + count results for a hash list with filters + """ + + filter_expression = construct_report_filter(report_filters) + + try: + report_count = session.query(Report) \ + .filter(Report.run_id == run_id) \ + .outerjoin(File, + and_(Report.file_id == File.id, + File.run_id == run_id)) \ + .outerjoin(BugPathEvent, + Report.end_bugevent == BugPathEvent.id) \ + .outerjoin(SuppressBug, + and_(SuppressBug.hash == Report.bug_id, + SuppressBug.run_id == run_id)) \ + .filter(Report.bug_id.in_(diff_hash_list)) \ + .filter(filter_expression) \ + .count() + + if report_count is None: + report_count = 0 + + return report_count + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, + msg) + + # ----------------------------------------------------------------------- + @timefunc + def getDiffResultCount(self, + base_run_id, + new_run_id, + diff_type, + report_filters): + """ + count the diff results + """ + + session = self.__session + base_line_hashes, new_check_hashes = \ + self.__get_hashes_for_diff(session, + base_run_id, + new_run_id) + run_id = None + diff_hashes = [] + + if diff_type == DiffType.NEW: + diff_hashes = list(new_check_hashes.difference(base_line_hashes)) + if not diff_hashes: + return 0 + run_id = new_run_id + + elif diff_type == DiffType.RESOLVED: + diff_hashes = list(base_line_hashes.difference(new_check_hashes)) + if not diff_hashes: + return 0 + run_id = base_run_id + + elif diff_type == DiffType.UNRESOLVED: + diff_hashes = list(base_line_hashes.intersection(new_check_hashes)) + if not diff_hashes: + return 0 + run_id = new_run_id + + else: + msg = 'Unsupported diff type: ' + str(diff_type) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, + msg) + + return self.__queryDiffResultsCount(session, + diff_hashes, + run_id, + report_filters) + + # ----------------------------------------------------------------------- + def __queryDiffResultTypes(self, + session, + diff_hash_list, + run_id, + report_filters): + """ + query and count results for a hash list with filters + """ + try: + filter_expression = construct_report_filter(report_filters) + + q = session.query(Report) \ + .filter(Report.run_id == run_id) \ + .outerjoin(File, + Report.file_id == File.id) \ + .outerjoin(BugPathEvent, + Report.end_bugevent == BugPathEvent.id) \ + .outerjoin(SuppressBug, + and_(SuppressBug.hash == Report.bug_id, + SuppressBug.run_id == run_id)) \ + .order_by(Report.checker_id) \ + .filter(Report.bug_id.in_(diff_hash_list)) \ + .filter(filter_expression) \ + .all() + + count_results = defaultdict(int) + result_reports = defaultdict() + + # count and filter out the results for the same checker_id + for r in q: + count_results[r.checker_id] += 1 + result_reports[r.checker_id] = r + + results = [] + for checker_id, res in result_reports.items(): + results.append(ReportDataTypeCount(res.checker_id, + res.severity, + count_results[res.checker_id])) + + # result count ascending + results = sorted(results, key=lambda rep: rep.count, reverse=True) + return results + + except sqlalchemy.exc.SQLAlchemyError as alchemy_ex: + msg = str(alchemy_ex) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, + msg) + + # ----------------------------------------------------------------------- + @timefunc + def getDiffResultTypes(self, + base_run_id, + new_run_id, + diff_type, + report_filters): + + session = self.__session + base_line_hashes, new_check_hashes = \ + self.__get_hashes_for_diff(session, + base_run_id, + new_run_id) + + run_id = None + diff_hashes = [] + + if diff_type == DiffType.NEW: + diff_hashes = list(new_check_hashes.difference(base_line_hashes)) + if not diff_hashes: + return diff_hashes + run_id = new_run_id + + elif diff_type == DiffType.RESOLVED: + diff_hashes = list(base_line_hashes.difference(new_check_hashes)) + if not diff_hashes: + return diff_hashes + run_id = base_run_id + + elif diff_type == DiffType.UNRESOLVED: + diff_hashes = list(base_line_hashes.intersection(new_check_hashes)) + if not diff_hashes: + return diff_hashes + run_id = new_run_id + + else: + msg = 'Unsupported diff type: ' + str(diff_type) + LOG.error(msg) + raise shared.ttypes.RequestFailed(shared.ttypes.ErrorCode.DATABASE, + msg) + + return self.__queryDiffResultTypes(session, + diff_hashes, + run_id, + report_filters) Index: tools/codechecker/libcodechecker/viewer_server/client_db_access_server.py =================================================================== --- /dev/null +++ tools/codechecker/libcodechecker/viewer_server/client_db_access_server.py @@ -0,0 +1,223 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +''' +Main viewer server starts a http server which handles thrift clienta +and browser requests +''' +import os +import posixpath +import urllib +import errno +import socket +from multiprocessing.pool import ThreadPool + +from sqlalchemy.orm import sessionmaker +from sqlalchemy.orm import scoped_session + +from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler +from SimpleHTTPServer import SimpleHTTPRequestHandler + +from thrift.transport import TTransport +from thrift.protocol import TJSONProtocol + +from codeCheckerDBAccess import codeCheckerDBAccess + +from client_db_access_handler import ThriftRequestHandler + +from libcodechecker import logger +from libcodechecker import database_handler + +LOG = logger.get_new_logger('DB ACCESS') + + +class RequestHander(SimpleHTTPRequestHandler): + ''' + Handle thrift and browser requests + Simply modified and extended version of SimpleHTTPRequestHandler + ''' + + def __init__(self, request, client_address, server): + + self.sc_session = server.sc_session + + self.db_version_info = server.db_version_info + + BaseHTTPRequestHandler.__init__(self, + request, + client_address, + server) + + def log_message(self, msg_format, *args): + ''' silenting http server ''' + return + + def do_POST(self): + ''' handling thrift messages ''' + client_host, client_port = self.client_address + LOG.debug('Processing request from: ' + + client_host + ':' + + str(client_port)) + + # create new thrift handler + checker_md_docs = self.server.checker_md_docs + checker_md_docs_map = self.server.checker_md_docs_map + suppress_handler = self.server.suppress_handler + + protocol_factory = TJSONProtocol.TJSONProtocolFactory() + input_protocol_factory = protocol_factory + output_protocol_factory = protocol_factory + + itrans = TTransport.TFileObjectTransport(self.rfile) + otrans = TTransport.TFileObjectTransport(self.wfile) + itrans = TTransport.TBufferedTransport(itrans, + int(self.headers['Content-Length'])) + otrans = TTransport.TMemoryBuffer() + + iprot = input_protocol_factory.getProtocol(itrans) + oprot = output_protocol_factory.getProtocol(otrans) + + try: + session = self.sc_session() + acc_handler = ThriftRequestHandler(session, + checker_md_docs, + checker_md_docs_map, + suppress_handler, + self.db_version_info) + + processor = codeCheckerDBAccess.Processor(acc_handler) + processor.process(iprot, oprot) + result = otrans.getvalue() + + self.sc_session.remove() + + self.send_response(200) + self.send_header("content-type", "application/x-thrift") + self.send_header("Content-Length", len(result)) + self.end_headers() + self.wfile.write(result) + return + + except Exception as exn: + LOG.error(str(exn)) + self.send_error(404, "Request failed.") + # self.send_header("content-type", "application/x-thrift") + # self.end_headers() + # self.wfile.write('') + return + + def list_directory(self, path): + ''' disable directory listing ''' + self.send_error(404, "No permission to list directory") + return None + + def translate_path(self, path): + ''' + modified version from SimpleHTTPRequestHandler + path is set to www_root + + ''' + # abandon query parameters + path = path.split('?', 1)[0] + path = path.split('#', 1)[0] + path = posixpath.normpath(urllib.unquote(path)) + words = path.split('/') + words = filter(None, words) + path = self.server.www_root + for word in words: + drive, word = os.path.splitdrive(word) + head, word = os.path.split(word) + if word in (os.curdir, os.pardir): + continue + path = os.path.join(path, word) + return path + + +class CCSimpleHttpServer(HTTPServer): + ''' + Simple http server to handle requests from the clients + ''' + + daemon_threads = False + + def __init__(self, + server_address, + RequestHandlerClass, + db_conn_string, + pckg_data, + suppress_handler, + db_version_info): + + LOG.debug('Initializing HTTP server') + + self.www_root = pckg_data['www_root'] + self.doc_root = pckg_data['doc_root'] + self.checker_md_docs = pckg_data['checker_md_docs'] + self.checker_md_docs_map = pckg_data['checker_md_docs_map'] + self.suppress_handler = suppress_handler + self.db_version_info = db_version_info + self.__engine =database_handler.SQLServer.create_engine(db_conn_string) + + Session = scoped_session(sessionmaker()) + Session.configure(bind=self.__engine) + self.sc_session = Session + + self.__request_handlers = ThreadPool(processes=10) + + HTTPServer.__init__(self, server_address, + RequestHandlerClass, + bind_and_activate=True) + + def process_request_thread(self, request, client_address): + try: + # finish_request instatiates request handler class + self.finish_request(request, client_address) + self.shutdown_request(request) + except socket.error as serr: + if serr[0] == errno.EPIPE: + LOG.debug('Broken pipe') + LOG.debug(serr) + self.shutdown_request(request) + + except Exception as ex: + LOG.debug(ex) + self.handle_error(request, client_address) + self.shutdown_request(request) + + def process_request(self, request, client_address): + # sock_name = request.getsockname() + # LOG.debug('PROCESSING request: '+str(sock_name)+' from: ' + # +str(client_address)) + self.__request_handlers.apply_async(self.process_request_thread, + (request, client_address)) + + +def start_server(package_data, port, db_conn_string, suppress_handler, + not_host_only, db_version_info): + ''' + start http server to handle web client and thrift requests + ''' + LOG.debug('Starting the codechecker DB access server') + + if not_host_only: + access_server_host = '' + else: + access_server_host = 'localhost' + + LOG.debug('Suppressing to ' + str(suppress_handler.suppress_file)) + + server_addr = (access_server_host, port) + + http_server = CCSimpleHttpServer(server_addr, + RequestHander, + db_conn_string, + package_data, + suppress_handler, + db_version_info) + + LOG.info('Waiting for client requests on [' + + access_server_host + ':' + str(port) + ']') + http_server.serve_forever() + LOG.info('done.') Index: tools/codechecker/requirements.txt =================================================================== --- /dev/null +++ tools/codechecker/requirements.txt @@ -0,0 +1,5 @@ +sqlalchemy>=1.0.9 +alembic>=0.8.2 +psycopg2>=2.5.4 +pg8000>=1.10.2 +thrift>=0.9.1 Index: tools/codechecker/tests/unit/__init__.py =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/__init__.py @@ -0,0 +1,5 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. Index: tools/codechecker/tests/unit/plist_test_files/clang-3.4.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/clang-3.4.plist @@ -0,0 +1,633 @@ + + + + + clang_version +Ubuntu clang version 3.4.2-13ubuntu2 (tags/RELEASE_34/dot2-final) (based on LLVM 3.4.2) + files + + test.cpp + ./test.h + + diagnostics + + + path + + + kindcontrol + edges + + + start + + + line19 + col5 + file0 + + + line19 + col7 + file0 + + + end + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + + + + + kindevent + location + + line20 + col5 + file0 + + ranges + + + + line20 + col5 + file0 + + + line20 + col12 + file0 + + + + depth0 + extended_message + 'base' initialized to 0 + message + 'base' initialized to 0 + + + kindcontrol + edges + + + start + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + end + + + line21 + col5 + file0 + + + line21 + col13 + file0 + + + + + + + kindevent + location + + line21 + col15 + file0 + + ranges + + + + line21 + col15 + file0 + + + line21 + col18 + file0 + + + + depth0 + extended_message + Passing the value 0 via 1st parameter 'base' + message + Passing the value 0 via 1st parameter 'base' + + + kindevent + location + + line21 + col5 + file0 + + ranges + + + + line21 + col5 + file0 + + + line21 + col19 + file0 + + + + depth0 + extended_message + Calling 'test_func' + message + Calling 'test_func' + + + kindevent + location + + line6 + col1 + file0 + + depth1 + extended_message + Entered call from 'main' + message + Entered call from 'main' + + + kindcontrol + edges + + + start + + + line6 + col1 + file0 + + + line6 + col4 + file0 + + + end + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + + + + + kindcontrol + edges + + + start + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + end + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + + + + + kindcontrol + edges + + + start + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + end + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + + + + kindevent + location + + line8 + col22 + file0 + + ranges + + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + depth1 + extended_message + Passing the value 0 via 1st parameter 'num' + message + Passing the value 0 via 1st parameter 'num' + + + kindcontrol + edges + + + start + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + end + + + line8 + col10 + file0 + + + line8 + col20 + file0 + + + + + + + kindevent + location + + line8 + col10 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth1 + extended_message + Calling 'generate_id' + message + Calling 'generate_id' + + + kindevent + location + + line6 + col1 + file1 + + depth2 + extended_message + Entered call from 'test_func' + message + Entered call from 'test_func' + + + kindcontrol + edges + + + start + + + line6 + col1 + file1 + + + line6 + col3 + file1 + + + end + + + line7 + col14 + file1 + + + line7 + col14 + file1 + + + + + + + kindevent + location + + line7 + col14 + file1 + + ranges + + + + line7 + col12 + file1 + + + line7 + col17 + file1 + + + + depth2 + extended_message + Division by zero + message + Division by zero + + + descriptionDivision by zero + categoryLogic error + typeDivision by zero + issue_context_kindfunction + issue_contextgenerate_id + issue_hash1 + location + + line7 + col14 + file1 + + + + path + + + kindevent + location + + line8 + col5 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth0 + extended_message + Value stored to 'id' is never read + message + Value stored to 'id' is never read + + + descriptionValue stored to 'id' is never read + categoryDead store + typeDead assignment + issue_context_kindfunction + issue_contexttest_func + issue_hash2 + location + + line8 + col5 + file0 + + + + path + + + kindcontrol + edges + + + start + + + line14 + col3 + file0 + + + line14 + col6 + file0 + + + end + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + + + + + kindcontrol + edges + + + start + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + end + + + line16 + col1 + file0 + + + line16 + col1 + file0 + + + + + + + kindevent + location + + line16 + col1 + file0 + + ranges + + + + line14 + col3 + file0 + + + line14 + col29 + file0 + + + + depth0 + extended_message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + + + descriptionAddress of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + categoryLogic error + typeStack address stored into global variable + issue_context_kindfunction + issue_contexttest + issue_hash3 + location + + line16 + col1 + file0 + + + + + \ No newline at end of file Index: tools/codechecker/tests/unit/plist_test_files/clang-3.5.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/clang-3.5.plist @@ -0,0 +1,633 @@ + + + + + clang_version +Ubuntu clang version 3.5.2-2 (tags/RELEASE_352/final) (based on LLVM 3.5.2) + files + + test.cpp + ./test.h + + diagnostics + + + path + + + kindcontrol + edges + + + start + + + line19 + col5 + file0 + + + line19 + col7 + file0 + + + end + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + + + + + kindevent + location + + line20 + col5 + file0 + + ranges + + + + line20 + col5 + file0 + + + line20 + col12 + file0 + + + + depth0 + extended_message + 'base' initialized to 0 + message + 'base' initialized to 0 + + + kindcontrol + edges + + + start + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + end + + + line21 + col5 + file0 + + + line21 + col13 + file0 + + + + + + + kindevent + location + + line21 + col15 + file0 + + ranges + + + + line21 + col15 + file0 + + + line21 + col18 + file0 + + + + depth0 + extended_message + Passing the value 0 via 1st parameter 'base' + message + Passing the value 0 via 1st parameter 'base' + + + kindevent + location + + line21 + col5 + file0 + + ranges + + + + line21 + col5 + file0 + + + line21 + col19 + file0 + + + + depth0 + extended_message + Calling 'test_func' + message + Calling 'test_func' + + + kindevent + location + + line6 + col1 + file0 + + depth1 + extended_message + Entered call from 'main' + message + Entered call from 'main' + + + kindcontrol + edges + + + start + + + line6 + col1 + file0 + + + line6 + col4 + file0 + + + end + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + + + + + kindcontrol + edges + + + start + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + end + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + + + + + kindcontrol + edges + + + start + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + end + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + + + + kindevent + location + + line8 + col22 + file0 + + ranges + + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + depth1 + extended_message + Passing the value 0 via 1st parameter 'num' + message + Passing the value 0 via 1st parameter 'num' + + + kindcontrol + edges + + + start + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + end + + + line8 + col10 + file0 + + + line8 + col20 + file0 + + + + + + + kindevent + location + + line8 + col10 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth1 + extended_message + Calling 'generate_id' + message + Calling 'generate_id' + + + kindevent + location + + line6 + col1 + file1 + + depth2 + extended_message + Entered call from 'test_func' + message + Entered call from 'test_func' + + + kindcontrol + edges + + + start + + + line6 + col1 + file1 + + + line6 + col3 + file1 + + + end + + + line7 + col14 + file1 + + + line7 + col14 + file1 + + + + + + + kindevent + location + + line7 + col14 + file1 + + ranges + + + + line7 + col12 + file1 + + + line7 + col17 + file1 + + + + depth2 + extended_message + Division by zero + message + Division by zero + + + descriptionDivision by zero + categoryLogic error + typeDivision by zero + issue_context_kindfunction + issue_contextgenerate_id + issue_hash1 + location + + line7 + col14 + file1 + + + + path + + + kindevent + location + + line8 + col5 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth0 + extended_message + Value stored to 'id' is never read + message + Value stored to 'id' is never read + + + descriptionValue stored to 'id' is never read + categoryDead store + typeDead assignment + issue_context_kindfunction + issue_contexttest_func + issue_hash2 + location + + line8 + col5 + file0 + + + + path + + + kindcontrol + edges + + + start + + + line14 + col3 + file0 + + + line14 + col6 + file0 + + + end + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + + + + + kindcontrol + edges + + + start + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + end + + + line16 + col1 + file0 + + + line16 + col1 + file0 + + + + + + + kindevent + location + + line16 + col1 + file0 + + ranges + + + + line14 + col3 + file0 + + + line14 + col29 + file0 + + + + depth0 + extended_message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + + + descriptionAddress of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + categoryLogic error + typeStack address stored into global variable + issue_context_kindfunction + issue_contexttest + issue_hash3 + location + + line16 + col1 + file0 + + + + + \ No newline at end of file Index: tools/codechecker/tests/unit/plist_test_files/clang-3.6.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/clang-3.6.plist @@ -0,0 +1,633 @@ + + + + + clang_version +Ubuntu clang version 3.6.2-1 (tags/RELEASE_362/final) (based on LLVM 3.6.2) + files + + test.cpp + ./test.h + + diagnostics + + + path + + + kindcontrol + edges + + + start + + + line19 + col5 + file0 + + + line19 + col7 + file0 + + + end + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + + + + + kindevent + location + + line20 + col5 + file0 + + ranges + + + + line20 + col5 + file0 + + + line20 + col12 + file0 + + + + depth0 + extended_message + 'base' initialized to 0 + message + 'base' initialized to 0 + + + kindcontrol + edges + + + start + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + end + + + line21 + col5 + file0 + + + line21 + col13 + file0 + + + + + + + kindevent + location + + line21 + col15 + file0 + + ranges + + + + line21 + col15 + file0 + + + line21 + col18 + file0 + + + + depth0 + extended_message + Passing the value 0 via 1st parameter 'base' + message + Passing the value 0 via 1st parameter 'base' + + + kindevent + location + + line21 + col5 + file0 + + ranges + + + + line21 + col5 + file0 + + + line21 + col19 + file0 + + + + depth0 + extended_message + Calling 'test_func' + message + Calling 'test_func' + + + kindevent + location + + line6 + col1 + file0 + + depth1 + extended_message + Entered call from 'main' + message + Entered call from 'main' + + + kindcontrol + edges + + + start + + + line6 + col1 + file0 + + + line6 + col4 + file0 + + + end + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + + + + + kindcontrol + edges + + + start + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + end + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + + + + + kindcontrol + edges + + + start + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + end + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + + + + kindevent + location + + line8 + col22 + file0 + + ranges + + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + depth1 + extended_message + Passing the value 0 via 1st parameter 'num' + message + Passing the value 0 via 1st parameter 'num' + + + kindcontrol + edges + + + start + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + end + + + line8 + col10 + file0 + + + line8 + col20 + file0 + + + + + + + kindevent + location + + line8 + col10 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth1 + extended_message + Calling 'generate_id' + message + Calling 'generate_id' + + + kindevent + location + + line6 + col1 + file1 + + depth2 + extended_message + Entered call from 'test_func' + message + Entered call from 'test_func' + + + kindcontrol + edges + + + start + + + line6 + col1 + file1 + + + line6 + col3 + file1 + + + end + + + line7 + col14 + file1 + + + line7 + col14 + file1 + + + + + + + kindevent + location + + line7 + col14 + file1 + + ranges + + + + line7 + col12 + file1 + + + line7 + col17 + file1 + + + + depth2 + extended_message + Division by zero + message + Division by zero + + + descriptionDivision by zero + categoryLogic error + typeDivision by zero + issue_context_kindfunction + issue_contextgenerate_id + issue_hash1 + location + + line7 + col14 + file1 + + + + path + + + kindevent + location + + line8 + col5 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth0 + extended_message + Value stored to 'id' is never read + message + Value stored to 'id' is never read + + + descriptionValue stored to 'id' is never read + categoryDead store + typeDead assignment + issue_context_kindfunction + issue_contexttest_func + issue_hash2 + location + + line8 + col5 + file0 + + + + path + + + kindcontrol + edges + + + start + + + line14 + col3 + file0 + + + line14 + col6 + file0 + + + end + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + + + + + kindcontrol + edges + + + start + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + end + + + line16 + col1 + file0 + + + line16 + col1 + file0 + + + + + + + kindevent + location + + line16 + col1 + file0 + + ranges + + + + line14 + col3 + file0 + + + line14 + col29 + file0 + + + + depth0 + extended_message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + + + descriptionAddress of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + categoryLogic error + typeStack address stored into global variable + issue_context_kindfunction + issue_contexttest + issue_hash3 + location + + line16 + col1 + file0 + + + + + \ No newline at end of file Index: tools/codechecker/tests/unit/plist_test_files/clang-3.7-noerror.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/clang-3.7-noerror.plist @@ -0,0 +1,14 @@ + + + + + clang_version +Ubuntu clang version 3.7.0-2ubuntu1 (tags/RELEASE_370/final) (based on LLVM 3.7.0) + files + + + diagnostics + + + + \ No newline at end of file Index: tools/codechecker/tests/unit/plist_test_files/clang-3.7.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/clang-3.7.plist @@ -0,0 +1,636 @@ + + + + + clang_version +Ubuntu clang version 3.7.0-2ubuntu1 (tags/RELEASE_370/final) (based on LLVM 3.7.0) + files + + test.cpp + ./test.h + + diagnostics + + + path + + + kindcontrol + edges + + + start + + + line19 + col5 + file0 + + + line19 + col7 + file0 + + + end + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + + + + + kindevent + location + + line20 + col5 + file0 + + ranges + + + + line20 + col5 + file0 + + + line20 + col12 + file0 + + + + depth0 + extended_message + 'base' initialized to 0 + message + 'base' initialized to 0 + + + kindcontrol + edges + + + start + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + end + + + line21 + col5 + file0 + + + line21 + col13 + file0 + + + + + + + kindevent + location + + line21 + col15 + file0 + + ranges + + + + line21 + col15 + file0 + + + line21 + col18 + file0 + + + + depth0 + extended_message + Passing the value 0 via 1st parameter 'base' + message + Passing the value 0 via 1st parameter 'base' + + + kindevent + location + + line21 + col5 + file0 + + ranges + + + + line21 + col5 + file0 + + + line21 + col19 + file0 + + + + depth0 + extended_message + Calling 'test_func' + message + Calling 'test_func' + + + kindevent + location + + line6 + col1 + file0 + + depth1 + extended_message + Entered call from 'main' + message + Entered call from 'main' + + + kindcontrol + edges + + + start + + + line6 + col1 + file0 + + + line6 + col4 + file0 + + + end + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + + + + + kindcontrol + edges + + + start + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + end + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + + + + + kindcontrol + edges + + + start + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + end + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + + + + kindevent + location + + line8 + col22 + file0 + + ranges + + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + depth1 + extended_message + Passing the value 0 via 1st parameter 'num' + message + Passing the value 0 via 1st parameter 'num' + + + kindcontrol + edges + + + start + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + end + + + line8 + col10 + file0 + + + line8 + col20 + file0 + + + + + + + kindevent + location + + line8 + col10 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth1 + extended_message + Calling 'generate_id' + message + Calling 'generate_id' + + + kindevent + location + + line6 + col1 + file1 + + depth2 + extended_message + Entered call from 'test_func' + message + Entered call from 'test_func' + + + kindcontrol + edges + + + start + + + line6 + col1 + file1 + + + line6 + col3 + file1 + + + end + + + line7 + col14 + file1 + + + line7 + col14 + file1 + + + + + + + kindevent + location + + line7 + col14 + file1 + + ranges + + + + line7 + col12 + file1 + + + line7 + col17 + file1 + + + + depth2 + extended_message + Division by zero + message + Division by zero + + + descriptionDivision by zero + categoryLogic error + typeDivision by zero + check_namecore.DivideZero + issue_context_kindfunction + issue_contextgenerate_id + issue_hash1 + location + + line7 + col14 + file1 + + + + path + + + kindevent + location + + line8 + col5 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth0 + extended_message + Value stored to 'id' is never read + message + Value stored to 'id' is never read + + + descriptionValue stored to 'id' is never read + categoryDead store + typeDead assignment + check_namedeadcode.DeadStores + issue_context_kindfunction + issue_contexttest_func + issue_hash2 + location + + line8 + col5 + file0 + + + + path + + + kindcontrol + edges + + + start + + + line14 + col3 + file0 + + + line14 + col6 + file0 + + + end + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + + + + + kindcontrol + edges + + + start + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + end + + + line16 + col1 + file0 + + + line16 + col1 + file0 + + + + + + + kindevent + location + + line16 + col1 + file0 + + ranges + + + + line14 + col3 + file0 + + + line14 + col29 + file0 + + + + depth0 + extended_message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + + + descriptionAddress of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + categoryLogic error + typeStack address stored into global variable + check_namecore.StackAddressEscape + issue_context_kindfunction + issue_contexttest + issue_hash3 + location + + line16 + col1 + file0 + + + + + \ No newline at end of file Index: tools/codechecker/tests/unit/plist_test_files/clang-3.8-trunk.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/clang-3.8-trunk.plist @@ -0,0 +1,642 @@ + + + + + clang_version +clang version 3.8.0 (trunk 251510) + files + + test.cpp + ./test.h + + diagnostics + + + path + + + kindcontrol + edges + + + start + + + line19 + col5 + file0 + + + line19 + col7 + file0 + + + end + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + + + + + kindevent + location + + line20 + col5 + file0 + + ranges + + + + line20 + col5 + file0 + + + line20 + col12 + file0 + + + + depth0 + extended_message + 'base' initialized to 0 + message + 'base' initialized to 0 + + + kindcontrol + edges + + + start + + + line20 + col5 + file0 + + + line20 + col7 + file0 + + + end + + + line21 + col5 + file0 + + + line21 + col13 + file0 + + + + + + + kindevent + location + + line21 + col15 + file0 + + ranges + + + + line21 + col15 + file0 + + + line21 + col18 + file0 + + + + depth0 + extended_message + Passing the value 0 via 1st parameter 'base' + message + Passing the value 0 via 1st parameter 'base' + + + kindevent + location + + line21 + col5 + file0 + + ranges + + + + line21 + col5 + file0 + + + line21 + col19 + file0 + + + + depth0 + extended_message + Calling 'test_func' + message + Calling 'test_func' + + + kindevent + location + + line6 + col1 + file0 + + depth1 + extended_message + Entered call from 'main' + message + Entered call from 'main' + + + kindcontrol + edges + + + start + + + line6 + col1 + file0 + + + line6 + col4 + file0 + + + end + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + + + + + kindcontrol + edges + + + start + + + line7 + col5 + file0 + + + line7 + col7 + file0 + + + end + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + + + + + kindcontrol + edges + + + start + + + line8 + col5 + file0 + + + line8 + col6 + file0 + + + end + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + + + + kindevent + location + + line8 + col22 + file0 + + ranges + + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + + depth1 + extended_message + Passing the value 0 via 1st parameter 'num' + message + Passing the value 0 via 1st parameter 'num' + + + kindcontrol + edges + + + start + + + line8 + col22 + file0 + + + line8 + col25 + file0 + + + end + + + line8 + col10 + file0 + + + line8 + col20 + file0 + + + + + + + kindevent + location + + line8 + col10 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth1 + extended_message + Calling 'generate_id' + message + Calling 'generate_id' + + + kindevent + location + + line6 + col1 + file1 + + depth2 + extended_message + Entered call from 'test_func' + message + Entered call from 'test_func' + + + kindcontrol + edges + + + start + + + line6 + col1 + file1 + + + line6 + col3 + file1 + + + end + + + line7 + col14 + file1 + + + line7 + col14 + file1 + + + + + + + kindevent + location + + line7 + col14 + file1 + + ranges + + + + line7 + col12 + file1 + + + line7 + col17 + file1 + + + + depth2 + extended_message + Division by zero + message + Division by zero + + + descriptionDivision by zero + categoryLogic error + typeDivision by zero + check_namecore.DivideZero + + issue_hash_content_of_line_in_context79e31a6ba028f0b7d9779faf4a6cb9cf + issue_context_kindfunction + issue_contextgenerate_id + issue_hash_function_offset1 + location + + line7 + col14 + file1 + + + + path + + + kindevent + location + + line8 + col5 + file0 + + ranges + + + + line8 + col10 + file0 + + + line8 + col26 + file0 + + + + depth0 + extended_message + Value stored to 'id' is never read + message + Value stored to 'id' is never read + + + descriptionValue stored to 'id' is never read + categoryDead store + typeDead assignment + check_namedeadcode.DeadStores + + issue_hash_content_of_line_in_context8714f42d8328bc78d5d7bff6ced918cc + issue_context_kindfunction + issue_contexttest_func + issue_hash_function_offset2 + location + + line8 + col5 + file0 + + + + path + + + kindcontrol + edges + + + start + + + line14 + col3 + file0 + + + line14 + col6 + file0 + + + end + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + + + + + kindcontrol + edges + + + start + + + line15 + col3 + file0 + + + line15 + col3 + file0 + + + end + + + line16 + col1 + file0 + + + line16 + col1 + file0 + + + + + + + kindevent + location + + line16 + col1 + file0 + + ranges + + + + line14 + col3 + file0 + + + line14 + col29 + file0 + + + + depth0 + extended_message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + message + Address of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + + + descriptionAddress of stack memory associated with local variable 'str' is still referred to by the global variable 'p' upon returning to the caller. This will be a dangling reference + categoryLogic error + typeStack address stored into global variable + check_namecore.StackAddressEscape + + issue_hash_content_of_line_in_contextf7b5072d428e890f2d309217f3ead16f + issue_context_kindfunction + issue_contexttest + issue_hash_function_offset3 + location + + line16 + col1 + file0 + + + + + \ No newline at end of file Index: tools/codechecker/tests/unit/plist_test_files/gen_plist/README.md =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/gen_plist/README.md @@ -0,0 +1,9 @@ +## Generate plist files + +Use these scripts and source files to generate plist report files. + +call gen_plist script with the clang binary and the plist output filename +Eg.: +~~~~~~~~~~~ +./gen_plist clang-3.7 clang-3.7.plist +~~~~~~~~~~~ Index: tools/codechecker/tests/unit/plist_test_files/gen_plist/gen_noerror_plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/gen_plist/gen_noerror_plist @@ -0,0 +1,12 @@ +#!/bin/bash +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +set -x # echo on + +CLANG=$1 +OUT=$2 +$CLANG --analyze -Xclang -analyzer-opt-analyze-headers -Xclang -analyzer-output=plist-multi-file -o $OUT noerror.cpp Index: tools/codechecker/tests/unit/plist_test_files/gen_plist/gen_plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/gen_plist/gen_plist @@ -0,0 +1,12 @@ +#!/bin/bash +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +set -x # echo on + +CLANG=$1 +OUT=$2 +$CLANG --analyze -Xclang -analyzer-opt-analyze-headers -Xclang -analyzer-output=plist-multi-file -o $OUT test.cpp -I. Index: tools/codechecker/tests/unit/plist_test_files/gen_plist/noerror.cpp =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/gen_plist/noerror.cpp @@ -0,0 +1,10 @@ +/* -*- coding: utf-8 -*- +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +*/ + +int main(){ + return 0; +} Index: tools/codechecker/tests/unit/plist_test_files/gen_plist/test.h =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/gen_plist/test.h @@ -0,0 +1,17 @@ +/* -*- coding: utf-8 -*- +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +*/ + +#ifndef TEST_H +#define TEST_H + +#include + +int generate_id(int num){ + return 42/num; // warn +} + +#endif // TEST_H Index: tools/codechecker/tests/unit/plist_test_files/gen_plist/test.cpp =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/plist_test_files/gen_plist/test.cpp @@ -0,0 +1,31 @@ +/* -*- coding: utf-8 -*- +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. +*/ + +#include +#include + +using namespace std; + +void test_func(int base){ + int id; + id = generate_id(base); +} + +char const *p; + +void test() { + char const str[] = "string"; + p = str; // warn +} + +int main(){ + int unused; // warn + int base = 0; + test_func(base); + test(); + return 0; +} Index: tools/codechecker/tests/unit/source_suppress_test_files/test_file_1 =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/source_suppress_test_files/test_file_1 @@ -0,0 +1,73 @@ +// THIS FILE IS ENCODED IN UTF-8! BE CAREFUL WITH MODIFYING IT + +int test_func(int num){ // line 3 + cout << "test func" << endl; + return 0; +} + +// this is an ordinary comment +int test_func(int num){ // line 9 + cout << "test func" << endl; + return 1; +} + +// this is another non checker related comment +// codechecker_suppress [all] some comment +void test_func(int num){ // line 16 + // suppress all checker results + cout << "test func" << endl; +} + +// codechecker_suppress [all] some long +// comment +void test_func(int num){ // line 23 + // suppress all checker results + cout << "test func" << endl; +} + +// codechecker_suppress [my_checker_1, my_checker_2] some comment +void test_func(int num){ // line 29 + // suppress some checker results my_checker_2 and my_checker_2 + cout << "test func" << endl; +} + +// codechecker_suppress [my.checker_1 my.checker_2] some really +// long comment +void test_func(int num){ // line 36 + cout << "test func" << endl; +} + +// codechecker_suppress [my.Checker_1, my.Checker_2] some really +// really +// long comment +void test_func(int num){ // line 43 + cout << "test func" << endl; +} + +// codechecker_suppress [my_checker_1, my_checker_2] some really +// really + +void test_func(int num){ // line 50 + // wrong formatted suppress comment + cout << "test func" << endl; +} + +// codechecker_suppress [my.checker_1, my.checker_2] +// i/';0 (*&^%$#@!) +void test_func(int num){ // line 57 + // wrong formatted suppress comment + cout << "test func" << endl; +} + +// codechecker_suppress [ my_checker_1 ] +// áúőóüöáé ▬▬▬▬▬▬▬▬▬▬ஜ۩۞۩ஜ▬▬▬▬▬▬▬▬▬▬ +void test_func(int num){ // line 64 + // wrong formatted suppress comment + cout << "test func" << endl; +} + +// codechecker_suppress [ my_checker_1 ] +void test_func(int num){ // line 70 + // wrong formatted suppress comment + cout << "test func" << endl; +} Index: tools/codechecker/tests/unit/source_suppress_test_files/test_file_2 =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/source_suppress_test_files/test_file_2 @@ -0,0 +1,5 @@ +// first line non checker comment +int test_func(int num){ // line 2 + cout << 'test func' << endl; + return 0; +} Index: tools/codechecker/tests/unit/test_buildcmd_escaping.py =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/test_buildcmd_escaping.py @@ -0,0 +1,146 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +"""Test the build commands escaping and execution.""" + +import json +import os +import shutil +import StringIO +import tempfile +import unittest +from contextlib import closing + +from libcodechecker import log_parser +from libcodechecker import build_manager +from libcodechecker.analyzers import analyzer_base + + +class BuildCmdTestNose(unittest.TestCase): + """ + Test the build command escaping and execution. + """ + + @classmethod + def setup_class(cls): + """ + Make a temporary directory and generate a source file + which will be built. + """ + cls.tmp_dir = tempfile.mkdtemp() + cls.src_file_name = "main.cpp" + cls.src_file_path = os.path.join(cls.tmp_dir, + cls.src_file_name) + cls.compiler = "g++" + + with open(cls.src_file_path, "w") as test_src: + test_src.write(""" + #include + + #ifndef MYPATH + #define MYPATH "/some/path" + #endif + + int main(){ + std::cout<< MYPATH << std::endl; + return 0; + }""") + + @classmethod + def teardown_class(cls): + """ + Clean temporary directory and files. + """ + dir_to_clean = cls.tmp_dir + shutil.rmtree(dir_to_clean) + + def __get_cmp_json(self, buildcmd): + """ + Generate a compile command json file. + """ + compile_cmds = [] + + compile_cmd = {} + compile_cmd["directory"] = self.tmp_dir + compile_cmd["command"] = buildcmd + " -c " + self.src_file_path + compile_cmd["file"] = self.src_file_path + + compile_cmds.append(compile_cmd) + return json.dumps(compile_cmds) + + def __get_comp_actions(self, compile_cmd): + """ + Generate a compilation command json file and parse it + to return the compilation actions. + """ + comp_cmd_json = self.__get_cmp_json(compile_cmd) + with closing(StringIO.StringIO()) as text: + text.write(comp_cmd_json) + return log_parser.parse_compile_commands_json(text) + + def test_buildmgr(self): + """ + Check some simple command to be executed by + the build manager. + """ + cmd = 'cd ' + self.tmp_dir + ' && echo "test"' + print("Running: " + cmd) + ret_val = build_manager.execute_buildcmd(cmd) + self.assertEqual(ret_val, 0) + + def test_analyzer_exec_double_quote(self): + """ + Test the process execution by the analyzer, + If the escaping fails the source file will not compile. + """ + compile_cmd = self.compiler + \ + ' -DDEBUG -DMYPATH="/this/some/path/"' + + comp_actions = self.__get_comp_actions(compile_cmd) + + for comp_action in comp_actions: + for source in comp_action.sources: + cmd = [self.compiler] + cmd.extend(comp_action.analyzer_options) + cmd.append(str(source)) + cwd = comp_action.directory + + print(cmd) + print(cwd) + + ret_val, stdout, stderr = analyzer_base.SourceAnalyzer \ + .run_proc(' '.join(cmd), cwd=cwd) + + print(stdout) + print(stderr) + self.assertEqual(ret_val, 0) + + def test_analyzer_ansic_double_quote(self): + """ + Test the process execution by the analyzer with ansi-C like + escape characters in it \". + If the escaping fails the source file will not compile. + """ + compile_cmd = self.compiler + ' -DMYPATH=\"/some/other/path\"' + comp_actions = self.__get_comp_actions(compile_cmd) + + for comp_action in comp_actions: + for source in comp_action.sources: + cmd = [self.compiler] + cmd.extend(comp_action.analyzer_options) + cmd.append(str(source)) + cwd = comp_action.directory + + print(cmd) + print(cwd) + + ret_val, stdout, stderr = analyzer_base.SourceAnalyzer \ + .run_proc(' '.join(cmd), cwd=cwd) + + print(stdout) + print(stderr) + + self.assertEqual(ret_val, 0) Index: tools/codechecker/tests/unit/test_plist_parser.py =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/test_plist_parser.py @@ -0,0 +1,209 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +"""Test the parsing of the plist generated by multiple clang versions.""" + +import os +import unittest + +from libcodechecker import plist_parser + + +class PlistParserTestCaseNose(unittest.TestCase): + """Test the parsing of the plist generated by multiple clang versions.""" + + @classmethod + def setup_class(cls): + """Initialize test source file.""" + # Bugs found by these checkers in the test source files. + cls.__found_checker_names = [ + 'core.DivideZero', + 'core.StackAddressEscape', + 'deadcode.DeadStores'] + + # Previous clang versions do not generate the checker name into the + # plist output. + cls.__not_found_checker_names = ['NOT FOUND'] + + # Bug id hash values generated by codechecker framework. + cls.__framework_generated_hashes = [ + 'e9fb5a280e64610cfa82472117c8d0ac', + '9536c46411df42db29014729867146fa', + 'b1bc0e8364a255659522055d1e15cd16'] + + # Bug were found in these test files. + cls.__found_file_names = ['test.cpp', './test.h'] + + # Already generated plist files for the tests. + cls.__this_dir = os.path.dirname(__file__) + cls.__plist_test_files = os.path.join( + cls.__this_dir, 'plist_test_files') + + def __core_div_bug_event(self, bug_event): + """Check the core.dividezero checker last event position.""" + self.assertEqual(bug_event.start_pos.line, 7) + self.assertEqual(bug_event.start_pos.col, 12) + self.assertEqual(bug_event.start_pos.file_path, './test.h') + self.assertEqual(bug_event.end_pos.line, 7) + self.assertEqual(bug_event.end_pos.col, 17) + self.assertEqual(bug_event.end_pos.file_path, './test.h') + + def __core_stack_addr_esc_event(self, bug_event): + """Check the core.StackAddressEscape checker last event position.""" + self.assertEqual(bug_event.start_pos.line, 14) + self.assertEqual(bug_event.start_pos.col, 3) + self.assertEqual(bug_event.start_pos.file_path, 'test.cpp') + self.assertEqual(bug_event.end_pos.line, 14) + self.assertEqual(bug_event.end_pos.col, 29) + self.assertEqual(bug_event.end_pos.file_path, 'test.cpp') + + def test_empty_file(self): + """Plist file is empty.""" + empty_plist = os.path.join(self.__plist_test_files, 'empty_file') + files, bugs = plist_parser.parse_plist(empty_plist) + self.assertEquals(files, []) + self.assertEquals(bugs, []) + + def test_no_bug_file(self): + """There was no bug in the checked file.""" + no_bug_plist = os.path.join( + self.__plist_test_files, 'clang-3.7-noerror.plist') + files, bugs = plist_parser.parse_plist(no_bug_plist) + self.assertEquals(files, []) + self.assertEquals(bugs, []) + + def test_clang34_plist(self): + """ + Check plist generated by clang 3.4 checker name is not in the plist + file plist parser tries to find out the name. + """ + clang34_plist = os.path.join( + self.__plist_test_files, 'clang-3.4.plist') + files, bugs = plist_parser.parse_plist(clang34_plist) + self.assertEquals(files, self.__found_file_names) + self.assertEquals(len(bugs), 3) + + valid_checker_names = [] + valid_checker_names.extend(self.__found_checker_names) + valid_checker_names.extend(self.__not_found_checker_names) + + for bug in bugs: + self.assertIn(bug.checker_name, valid_checker_names) + if bug.checker_name == 'core.DivideZero': + self.__core_div_bug_event(bug.get_last_event()) + if bug.checker_name == 'NOT FOUND': + self.__core_stack_addr_esc_event(bug.get_last_event()) + + self.assertIn(bug.hash_value, self.__framework_generated_hashes) + + def test_clang35_plist(self): + """ + Check plist generated by clang 3.5 checker name is not in the plist + file plist parser tries to find out the name. + """ + clang35_plist = os.path.join( + self.__plist_test_files, 'clang-3.5.plist') + files, bugs = plist_parser.parse_plist(clang35_plist) + self.assertEquals(files, self.__found_file_names) + self.assertEquals(len(bugs), 3) + + valid_checker_names = [] + valid_checker_names.extend(self.__found_checker_names) + valid_checker_names.extend(self.__not_found_checker_names) + + for bug in bugs: + self.assertIn(bug.checker_name, valid_checker_names) + if bug.checker_name == 'core.DivideZero': + self.__core_div_bug_event(bug.get_last_event()) + if bug.checker_name == 'NOT FOUND': + self.__core_stack_addr_esc_event(bug.get_last_event()) + + self.assertIn(bug.hash_value, self.__framework_generated_hashes) + + def test_clang36_plist(self): + """ + Check plist generated by clang 3.6 checker name is not in the plist + file plist parser tries to find out the name. + """ + clang36_plist = os.path.join( + self.__plist_test_files, 'clang-3.6.plist') + files, bugs = plist_parser.parse_plist(clang36_plist) + self.assertEquals(files, self.__found_file_names) + self.assertEquals(len(bugs), 3) + + valid_checker_names = [] + valid_checker_names.extend(self.__found_checker_names) + valid_checker_names.extend(self.__not_found_checker_names) + + for bug in bugs: + self.assertIn(bug.checker_name, valid_checker_names) + if bug.checker_name == 'core.DivideZero': + self.__core_div_bug_event(bug.get_last_event()) + if bug.checker_name == 'NOT FOUND': + self.__core_stack_addr_esc_event(bug.get_last_event()) + + self.assertIn(bug.hash_value, self.__framework_generated_hashes) + + def test_clang37_plist(self): + """ + Check plist generated by clang 3.7 checker name should be in the plist + file. + """ + clang37_plist = os.path.join( + self.__plist_test_files, 'clang-3.7.plist') + files, bugs = plist_parser.parse_plist(clang37_plist) + + self.assertEquals(files, self.__found_file_names) + self.assertEquals(len(bugs), 3) + + valid_checker_names = [] + valid_checker_names.extend(self.__found_checker_names) + + for bug in bugs: + # Checker name should be in the plist file. + self.assertNotEqual(bug.checker_name, 'NOT FOUND') + + self.assertIn(bug.checker_name, valid_checker_names) + + if bug.checker_name == 'core.DivideZero': + self.__core_div_bug_event(bug.get_last_event()) + if bug.checker_name == 'core.StackAddressEscape': + self.__core_stack_addr_esc_event(bug.get_last_event()) + + self.assertNotEquals(bug.hash_value, '') + + def test_clang38_trunk_plist(self): + """ + Check plist generated by clang 3.8 trunk checker name and bug hash + should be in the plist file. + """ + clang38_plist = os.path.join( + self.__plist_test_files, 'clang-3.8-trunk.plist') + files, bugs = plist_parser.parse_plist(clang38_plist) + + self.assertEquals(files, self.__found_file_names) + self.assertEquals(len(bugs), 3) + + valid_checker_names = [] + valid_checker_names.extend(self.__found_checker_names) + + for bug in bugs: + # Checker name should be in the plist file. + self.assertNotEqual(bug.checker_name, 'NOT FOUND') + + self.assertIn(bug.checker_name, valid_checker_names) + + if bug.checker_name == 'core.DivideZero': + self.__core_div_bug_event(bug.get_last_event()) + self.assertEquals( + bug.hash_value, '79e31a6ba028f0b7d9779faf4a6cb9cf') + if bug.checker_name == 'core.StackAddressEscape': + self.__core_stack_addr_esc_event(bug.get_last_event()) + self.assertEquals( + bug.hash_value, 'f7b5072d428e890f2d309217f3ead16f') + if bug.checker_name == 'deadcode.DeadStores': + self.assertEquals( + bug.hash_value, '8714f42d8328bc78d5d7bff6ced918cc') Index: tools/codechecker/tests/unit/test_source_suppress.py =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/test_source_suppress.py @@ -0,0 +1,133 @@ +# -*- coding: utf-8 -*- +# +# ----------------------------------------------------------------------------- +# The CodeChecker Infrastructure +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. +# ----------------------------------------------------------------------------- + +"""Tests for suppressing by comment in source file.""" + +import os +import unittest + +from libcodechecker.suppress_handler import SourceSuppressHandler + + +class SourceSuppressTestCase(unittest.TestCase): + """Tests for suppressing by comment in source file.""" + + @classmethod + def setup_class(cls): + """Initialize test source file references.""" + cls.__test_src_dir = os.path.join( + os.path.dirname(__file__), 'source_suppress_test_files') + + cls.__tmp_srcfile_1 = os.path.join(cls.__test_src_dir, 'test_file_1') + cls.__tmp_srcfile_2 = os.path.join(cls.__test_src_dir, 'test_file_2') + + def test_suppress_first_line(self): + """Bug is reported for the first line.""" + test_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 3) + res = test_handler.check_source_suppress() + self.assertFalse(res) + self.assertEqual(test_handler.suppressed_checkers(), []) + self.assertIsNone(test_handler.suppress_comment()) + + def test_no_comment(self): + """There is no comment above the bug line.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 9) + res = sp_handler.check_source_suppress() + self.assertFalse(res) + self.assertEqual(sp_handler.suppressed_checkers(), []) + self.assertIsNone(sp_handler.suppress_comment()) + + def test_no_suppress_comment(self): + """There is suppress comment above the bug line.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 16) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual(sp_handler.suppressed_checkers(), ['all']) + self.assertEqual(sp_handler.suppress_comment(), 'some comment') + + def test_multi_liner_all(self): + """There is suppress comment above the bug line.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 23) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual(sp_handler.suppressed_checkers(), ['all']) + self.assertEqual(sp_handler.suppress_comment(), 'some long comment') + + def test_one_liner_all(self): + """There is suppress comment above the bug line.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 29) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual( + sp_handler.suppressed_checkers(), ['my_checker_1', 'my_checker_2']) + self.assertEqual(sp_handler.suppress_comment(), 'some comment') + + def test_multi_liner_all_2(self): + """There is suppress comment above the bug line.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 36) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual( + sp_handler.suppressed_checkers(), ['my.checker_1', 'my.checker_2']) + self.assertEqual( + sp_handler.suppress_comment(), 'some really long comment') + + def test_one_liner_some_checkers(self): + """There is suppress comment above the bug line.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 43) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual( + sp_handler.suppressed_checkers(), ['my.Checker_1', 'my.Checker_2']) + self.assertEqual( + sp_handler.suppress_comment(), 'some really really long comment') + + def test_multi_liner_some_checkers(self): + """There is suppress comment above the bug line.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 50) + res = sp_handler.check_source_suppress() + self.assertFalse(res) + self.assertEqual(sp_handler.suppressed_checkers(), []) + self.assertIsNone(sp_handler.suppress_comment()) + + def test_comment_characters(self): + """Check for different special comment characters.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 57) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual( + sp_handler.suppressed_checkers(), ['my.checker_1', 'my.checker_2']) + self.assertEqual(sp_handler.suppress_comment(), "i/';0 (*&^%$#@!)") + + def test_fancy_comment_characters(self): + """Check fancy comment.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 64) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual(sp_handler.suppressed_checkers(), ['my_checker_1']) + self.assertEqual( + sp_handler.suppress_comment(), + "áúőóüöáé ▬▬▬▬▬▬▬▬▬▬ஜ۩۞۩ஜ▬▬▬▬▬▬▬▬▬▬") + + def test_no_fancy_comment(self): + """Check no fancy comment.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_1, 70) + res = sp_handler.check_source_suppress() + self.assertTrue(res) + self.assertEqual(sp_handler.suppressed_checkers(), ['my_checker_1']) + self.assertEqual( + sp_handler.suppress_comment(), + 'WARNING! suppress comment is missing') + + def test_malformed_commment_format(self): + """Check malformed comment.""" + sp_handler = SourceSuppressHandler(self.__tmp_srcfile_2, 1) + res = sp_handler.check_source_suppress() + self.assertFalse(res) + self.assertEqual(sp_handler.suppressed_checkers(), []) + self.assertIsNone(sp_handler.suppress_comment()) Index: tools/codechecker/tests/unit/test_tidy_output_converter.py =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/test_tidy_output_converter.py @@ -0,0 +1,253 @@ +# -*- coding: utf-8 -*- +# The LLVM Compiler Infrastructure +# +# This file is distributed under the University of Illinois Open Source +# License. See LICENSE.TXT for details. + +""" +This module tests the correctness of the OutputParser and PListConverter, which +used in sequence transform a Clang Tidy output file to a plist file. +""" + +import os +import unittest +import copy +import StringIO + +import libcodechecker.tidy_output_converter as tidy_out_conv + + +def setup_module(): + """Setup the test tidy reprs for the test classes in the module.""" + os.chdir(os.path.join(os.path.dirname(__file__), 'tidy_output_test_files')) + + # tidy1.out Message/Note representation + tidy1_repr = [ + tidy_out_conv.Message( + os.path.abspath('files/test.cpp'), + 8, 12, + 'Division by zero', + 'clang-analyzer-core.DivideZero', + None, + [tidy_out_conv.Note( + os.path.abspath('files/test.cpp'), + 8, 12, + 'Division by zero')]), + tidy_out_conv.Message( + os.path.abspath('files/test.cpp'), + 8, 12, + 'remainder by zero is undefined', + 'clang-diagnostic-division-by-zero') + ] + + # tidy2.out Message/Note representation + tidy2_repr = [ + tidy_out_conv.Message( + os.path.abspath('files/test2.cpp'), + 5, 7, + "unused variable 'y'", + 'clang-diagnostic-unused-variable'), + tidy_out_conv.Message( + os.path.abspath('files/test2.cpp'), + 13, 12, + 'Division by zero', + 'clang-analyzer-core.DivideZero', + None, + [ + tidy_out_conv.Note( + os.path.abspath('files/test2.cpp'), + 9, 7, + "Left side of '||' is false"), + tidy_out_conv.Note( + os.path.abspath('files/test2.cpp'), + 9, 3, + 'Taking false branch'), + tidy_out_conv.Note( + os.path.abspath('files/test2.cpp'), + 13, 12, + 'Division by zero') + ]), + tidy_out_conv.Message( + os.path.abspath('files/test2.cpp'), + 13, 12, + 'remainder by zero is undefined', + 'clang-diagnostic-division-by-zero'), + ] + + # tidy3.out Message/Note representation + tidy3_repr = [ + tidy_out_conv.Message( + os.path.abspath('files/test3.cpp'), + 4, 12, + 'use nullptr', + 'modernize-use-nullptr', + [tidy_out_conv.Note( + os.path.abspath('files/test3.cpp'), + 4, 12, + 'nullptr')]), + tidy_out_conv.Message( + os.path.abspath('files/test3.hh'), + 6, 6, + "Dereference of null pointer (loaded from variable 'x')", + 'clang-analyzer-core.NullDereference', + None, + [ + tidy_out_conv.Note( + os.path.abspath('files/test3.cpp'), + 4, 3, + "'x' initialized to a null pointer value"), + tidy_out_conv.Note( + os.path.abspath('files/test3.cpp'), + 6, 11, + "Assuming 'argc' is > 3"), + tidy_out_conv.Note( + os.path.abspath('files/test3.cpp'), + 6, 3, + 'Taking true branch'), + tidy_out_conv.Note( + os.path.abspath('files/test3.cpp'), + 7, 9, + "Passing null pointer value via 1st parameter 'x'"), + tidy_out_conv.Note( + os.path.abspath('files/test3.cpp'), + 7, 5, + "Calling 'bar'"), + tidy_out_conv.Note( + os.path.abspath('files/test3.hh'), + 6, 6, + "Dereference of null pointer (loaded from variable 'x')") + ]) + ] + + TidyOutputParserTestCase.tidy1_repr = tidy1_repr + TidyOutputParserTestCase.tidy2_repr = tidy2_repr + TidyOutputParserTestCase.tidy3_repr = tidy3_repr + TidyPListConverterTestCase.tidy1_repr = tidy1_repr + TidyPListConverterTestCase.tidy2_repr = tidy2_repr + TidyPListConverterTestCase.tidy3_repr = tidy3_repr + + +class TidyOutputParserTestCase(unittest.TestCase): + """ + Tests the output of the OutputParser, which converts a Clang Tidy output + file to zero or more tidy_output_converter.Message. + """ + + def setUp(self): + """Setup the OutputParser.""" + self.parser = tidy_out_conv.OutputParser() + + def test_absolute_path(self): + """Test for absolute paths in Messages.""" + for tfile in ['abs.out', 'tidy1.out']: + messages = self.parser.parse_messages_from_file(tfile) + self.assertNotEqual(len(messages), 0) + for message in messages: + self.assertTrue(os.path.isabs(message.path)) + + def test_empty1(self): + """Test an empty ClangTidy output file.""" + messages = self.parser.parse_messages_from_file('empty1.out') + self.assertEqual(messages, []) + + def test_empty2(self): + """Test a ClangTidy output file that only contains empty lines.""" + messages = self.parser.parse_messages_from_file('empty2.out') + self.assertEqual(messages, []) + + def test_tidy1(self): + """Test the generated Messages of tidy1.out ClangTidy output file.""" + messages = self.parser.parse_messages_from_file('tidy1.out') + self.assertEqual(len(messages), len(self.tidy1_repr)) + for message in messages: + self.assertIn(message, self.tidy1_repr) + + def test_tidy2(self): + """Test the generated Messages of tidy2.out ClangTidy output file.""" + messages = self.parser.parse_messages_from_file('tidy2.out') + self.assertEqual(len(messages), len(self.tidy2_repr)) + for message in messages: + self.assertIn(message, self.tidy2_repr) + + def test_tidy3(self): + """Test the generated Messages of tidy3.out ClangTidy output file.""" + messages = self.parser.parse_messages_from_file('tidy3.out') + self.assertEqual(len(messages), len(self.tidy3_repr)) + for message in messages: + self.assertIn(message, self.tidy3_repr) + + +class TidyPListConverterTestCase(unittest.TestCase): + """ + Test the output of the PListConverter, which converts Messages to plist + format. + """ + + def setUp(self): + """Setup the PListConverter.""" + self.plist_conv = tidy_out_conv.PListConverter() + + def test_empty(self): + """Test for empty Messages.""" + orig_plist = copy.deepcopy(self.plist_conv.plist) + + self.plist_conv.add_messages([]) + self.assertDictEqual(orig_plist, self.plist_conv.plist) + + output = StringIO.StringIO() + self.plist_conv.write(output) + + with open('empty.plist') as pfile: + exp = pfile.read() + self.assertEqual(exp, output.getvalue()) + + output.close() + + def test_tidy1(self): + """Test for the tidy1.plist file.""" + self.plist_conv.add_messages(self.tidy1_repr) + + # use relative path for this test + self.plist_conv.plist['files'][0] = 'files/test.cpp' + + output = StringIO.StringIO() + self.plist_conv.write(output) + + with open('tidy1.plist') as pfile: + exp = pfile.read() + self.assertEqual(exp, output.getvalue()) + + output.close() + + def test_tidy2(self): + """Test for the tidy2.plist file.""" + self.plist_conv.add_messages(self.tidy2_repr) + + # use relative path for this test + self.plist_conv.plist['files'][0] = 'files/test2.cpp' + + output = StringIO.StringIO() + self.plist_conv.write(output) + + with open('tidy2.plist') as pfile: + exp = pfile.read() + self.assertEqual(exp, output.getvalue()) + + output.close() + + def test_tidy3(self): + """Test for the tidy3.plist file.""" + self.plist_conv.add_messages(self.tidy3_repr) + + # use relative path for this test + self.plist_conv.plist['files'][0] = 'files/test3.cpp' + self.plist_conv.plist['files'][1] = 'files/test3.hh' + + output = StringIO.StringIO() + self.plist_conv.write(output) + + with open('tidy3.plist') as pfile: + exp = pfile.read() + self.assertEqual(exp, output.getvalue()) + + output.close() Index: tools/codechecker/tests/unit/tidy_output_test_files/abs.out =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/abs.out @@ -0,0 +1,9 @@ +/something/fake/test.cpp:8:12: warning: Division by zero [clang-analyzer-core.DivideZero] + return x % 0; + ^ +/something/fake/test.cpp:8:12: note: Division by zero + return x % 0; + ^ +/something/fake/test.cpp:8:12: warning: remainder by zero is undefined [clang-diagnostic-division-by-zero] + return x % 0; + ^ Index: tools/codechecker/tests/unit/tidy_output_test_files/empty.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/empty.plist @@ -0,0 +1,12 @@ + + + + + diagnostics + + + files + + + + Index: tools/codechecker/tests/unit/tidy_output_test_files/files/test.cpp =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/files/test.cpp @@ -0,0 +1,9 @@ +#include + +int main() { + int x; + + std::cin >> x; + + return x % 0; +} Index: tools/codechecker/tests/unit/tidy_output_test_files/files/test2.cpp =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/files/test2.cpp @@ -0,0 +1,14 @@ +#include + +int main() { + int x; + int y; + + std::cin >> x; + + if (false || x) { + return 42; + } + + return x % 0; +} Index: tools/codechecker/tests/unit/tidy_output_test_files/files/test3.hh =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/files/test3.hh @@ -0,0 +1,7 @@ +inline bool foo(bool arg) { + return false || arg; +} + +inline void bar(int* x) { + *x = 42; +} Index: tools/codechecker/tests/unit/tidy_output_test_files/files/test3.cpp =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/files/test3.cpp @@ -0,0 +1,10 @@ +#include "test3.hh" + +int main(int argc, const char** /*argv*/) { + int* x = 0; + + if (foo(argc > 3)) { + bar(x); + } + return 0; +} Index: tools/codechecker/tests/unit/tidy_output_test_files/tidy1.out =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/tidy1.out @@ -0,0 +1,9 @@ +files/test.cpp:8:12: warning: Division by zero [clang-analyzer-core.DivideZero] + return x % 0; + ^ +files/test.cpp:8:12: note: Division by zero + return x % 0; + ^ +files/test.cpp:8:12: warning: remainder by zero is undefined [clang-diagnostic-division-by-zero] + return x % 0; + ^ Index: tools/codechecker/tests/unit/tidy_output_test_files/tidy1.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/tidy1.plist @@ -0,0 +1,122 @@ + + + + + diagnostics + + + category + clang + check_name + clang-analyzer-core.DivideZero + description + Division by zero + location + + col + 12 + file + 0 + line + 8 + + path + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 8 + + message + Division by zero + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 8 + + message + Division by zero + + + edges + + + kind + control + + + type + clang-tidy + + + category + clang + check_name + clang-diagnostic-division-by-zero + description + remainder by zero is undefined + location + + col + 12 + file + 0 + line + 8 + + path + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 8 + + message + remainder by zero is undefined + + + edges + + + kind + control + + + type + clang-tidy + + + files + + files/test.cpp + + + Index: tools/codechecker/tests/unit/tidy_output_test_files/tidy2.out =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/tidy2.out @@ -0,0 +1,18 @@ +files/test2.cpp:5:7: warning: unused variable 'y' [clang-diagnostic-unused-variable] + int y; + ^ +files/test2.cpp:13:12: warning: Division by zero [clang-analyzer-core.DivideZero] + return x % 0; + ^ +files/test2.cpp:9:7: note: Left side of '||' is false + if (false || x) { + ^ +files/test2.cpp:9:3: note: Taking false branch + if (false || x) { + ^ +files/test2.cpp:13:12: note: Division by zero + return x % 0; + ^ +files/test2.cpp:13:12: warning: remainder by zero is undefined [clang-diagnostic-division-by-zero] + return x % 0; + ^ Index: tools/codechecker/tests/unit/tidy_output_test_files/tidy2.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/tidy2.plist @@ -0,0 +1,282 @@ + + + + + diagnostics + + + category + clang + check_name + clang-diagnostic-unused-variable + description + unused variable 'y' + location + + col + 7 + file + 0 + line + 5 + + path + + + depth + 0 + kind + event + location + + col + 7 + file + 0 + line + 5 + + message + unused variable 'y' + + + edges + + + kind + control + + + type + clang-tidy + + + category + clang + check_name + clang-analyzer-core.DivideZero + description + Division by zero + location + + col + 12 + file + 0 + line + 13 + + path + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 13 + + message + Division by zero + + + depth + 0 + kind + event + location + + col + 7 + file + 0 + line + 9 + + message + Left side of '||' is false + + + depth + 0 + kind + event + location + + col + 3 + file + 0 + line + 9 + + message + Taking false branch + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 13 + + message + Division by zero + + + edges + + + end + + + col + 3 + file + 0 + line + 9 + + + col + 3 + file + 0 + line + 9 + + + start + + + col + 7 + file + 0 + line + 9 + + + col + 7 + file + 0 + line + 9 + + + + + end + + + col + 12 + file + 0 + line + 13 + + + col + 12 + file + 0 + line + 13 + + + start + + + col + 3 + file + 0 + line + 9 + + + col + 3 + file + 0 + line + 9 + + + + + kind + control + + + type + clang-tidy + + + category + clang + check_name + clang-diagnostic-division-by-zero + description + remainder by zero is undefined + location + + col + 12 + file + 0 + line + 13 + + path + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 13 + + message + remainder by zero is undefined + + + edges + + + kind + control + + + type + clang-tidy + + + files + + files/test2.cpp + + + Index: tools/codechecker/tests/unit/tidy_output_test_files/tidy3.out =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/tidy3.out @@ -0,0 +1,25 @@ +files/test3.cpp:4:12: warning: use nullptr [modernize-use-nullptr] + int* x = 0; + ^ + nullptr +files/test3.hh:6:6: warning: Dereference of null pointer (loaded from variable 'x') [clang-analyzer-core.NullDereference] + *x = 42; + ^ +files/test3.cpp:4:3: note: 'x' initialized to a null pointer value + int* x = 0; + ^ +files/test3.cpp:6:11: note: Assuming 'argc' is > 3 + if (foo(argc > 3)) { + ^ +files/test3.cpp:6:3: note: Taking true branch + if (foo(argc > 3)) { + ^ +files/test3.cpp:7:9: note: Passing null pointer value via 1st parameter 'x' + bar(x); + ^ +files/test3.cpp:7:5: note: Calling 'bar' + bar(x); + ^ +files/test3.hh:6:6: note: Dereference of null pointer (loaded from variable 'x') + *x = 42; + ^ Index: tools/codechecker/tests/unit/tidy_output_test_files/tidy3.plist =================================================================== --- /dev/null +++ tools/codechecker/tests/unit/tidy_output_test_files/tidy3.plist @@ -0,0 +1,425 @@ + + + + + diagnostics + + + category + modernize + check_name + modernize-use-nullptr + description + use nullptr + location + + col + 12 + file + 0 + line + 4 + + path + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 4 + + message + use nullptr + + + depth + 0 + kind + event + location + + col + 12 + file + 0 + line + 4 + + message + nullptr (fixit) + + + edges + + + kind + control + + + type + clang-tidy + + + category + clang + check_name + clang-analyzer-core.NullDereference + description + Dereference of null pointer (loaded from variable 'x') + location + + col + 6 + file + 1 + line + 6 + + path + + + depth + 0 + kind + event + location + + col + 6 + file + 1 + line + 6 + + message + Dereference of null pointer (loaded from variable 'x') + + + depth + 0 + kind + event + location + + col + 3 + file + 0 + line + 4 + + message + 'x' initialized to a null pointer value + + + depth + 0 + kind + event + location + + col + 11 + file + 0 + line + 6 + + message + Assuming 'argc' is > 3 + + + depth + 0 + kind + event + location + + col + 3 + file + 0 + line + 6 + + message + Taking true branch + + + depth + 0 + kind + event + location + + col + 9 + file + 0 + line + 7 + + message + Passing null pointer value via 1st parameter 'x' + + + depth + 0 + kind + event + location + + col + 5 + file + 0 + line + 7 + + message + Calling 'bar' + + + depth + 0 + kind + event + location + + col + 6 + file + 1 + line + 6 + + message + Dereference of null pointer (loaded from variable 'x') + + + edges + + + end + + + col + 11 + file + 0 + line + 6 + + + col + 11 + file + 0 + line + 6 + + + start + + + col + 3 + file + 0 + line + 4 + + + col + 3 + file + 0 + line + 4 + + + + + end + + + col + 3 + file + 0 + line + 6 + + + col + 3 + file + 0 + line + 6 + + + start + + + col + 11 + file + 0 + line + 6 + + + col + 11 + file + 0 + line + 6 + + + + + end + + + col + 9 + file + 0 + line + 7 + + + col + 9 + file + 0 + line + 7 + + + start + + + col + 3 + file + 0 + line + 6 + + + col + 3 + file + 0 + line + 6 + + + + + end + + + col + 5 + file + 0 + line + 7 + + + col + 5 + file + 0 + line + 7 + + + start + + + col + 9 + file + 0 + line + 7 + + + col + 9 + file + 0 + line + 7 + + + + + end + + + col + 6 + file + 1 + line + 6 + + + col + 6 + file + 1 + line + 6 + + + start + + + col + 5 + file + 0 + line + 7 + + + col + 5 + file + 0 + line + 7 + + + + + kind + control + + + type + clang-tidy + + + files + + files/test3.cpp + files/test3.hh + + + Index: tools/codechecker/thrift_api/report_storage_server.thrift =================================================================== --- /dev/null +++ tools/codechecker/thrift_api/report_storage_server.thrift @@ -0,0 +1,104 @@ +// -*- coding: utf-8 -*- +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. + +include "shared.thrift" + +namespace py DBThriftAPI + +struct NeedFileResult { + 1: bool needed; + 2: i64 fileId; +} + +struct SuppressBugData { + 1: string bug_hash, + 2: string file_name, + 3: string comment +} +typedef list SuppressBugList + +// The order of the functions inditaces the order that must be maintained when +// calling into the server. +service CheckerReport { + // store checker run related data to the database + // by default updates the results if name was already found + // using the force flag removes existing analysis results for a run + i64 addCheckerRun( + 1: string command, + 2: string name, + 3: string version, + 4: bool force) + throws (1: shared.RequestFailed requestError), + + bool replaceConfigInfo( + 1: i64 run_id, + 2: shared.CheckerConfigList values) + throws (1: shared.RequestFailed requestError), + + bool addSuppressBug( + 1: i64 run_id, + 2: SuppressBugList bugsToSuppress + ) + throws (1: shared.RequestFailed requestError), + + # remove all suppress information from the database + bool cleanSuppressData( + 1: i64 run_id, + ) + throws (1: shared.RequestFailed requestError), + + # the map contains a path and a comment (can be empty) + bool addSkipPath( + 1: i64 run_id, + 2: map paths) + throws (1: shared.RequestFailed requestError), + + + // The next few following functions must be called via the same connection. + // ============================================================= + i64 addBuildAction( + 1: i64 run_id, + 2: string build_cmd, + 3: string check_cmd, + 4: string analyzer_type, + 5: string analyzed_source_file) + throws (1: shared.RequestFailed requestError), + + i64 addReport( + 1: i64 build_action_id, + 2: i64 file_id, + 3: string bug_hash, + 4: string checker_message, + 5: shared.BugPath bugpath, + 6: shared.BugPathEvents events, + 7: string checker_id, + 8: string checker_cat, + 9: string bug_type, + 10: shared.Severity severity, + 11: bool suppress) + throws (1: shared.RequestFailed requestError), + + bool finishBuildAction( + 1: i64 action_id, + 2: string failure) + throws (1: shared.RequestFailed requestError), + + NeedFileResult needFileContent( + 1: i64 run_id, + 2: string filepath) + throws (1: shared.RequestFailed requestError), + + bool addFileContent( + 1: i64 file_id, + 2: binary file_content) + throws (1: shared.RequestFailed requestError), + + bool finishCheckerRun(1: i64 run_id) + throws (1: shared.RequestFailed requestError), + + bool stopServer() + throws (1: shared.RequestFailed requestError) +} Index: tools/codechecker/thrift_api/report_viewer_server.thrift =================================================================== --- /dev/null +++ tools/codechecker/thrift_api/report_viewer_server.thrift @@ -0,0 +1,262 @@ +// -*- coding: utf-8 -*- +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. + +// ================================================= +// !!! Update version information if api changes!!! +// ================================================= +// backward incompatible changes should increase major version +// other changes should only increase minor version + +include "shared.thrift" + +namespace py codeCheckerDBAccess +namespace js codeCheckerDBAccess +namespace cpp cc.service.codechecker + +//================================================= +const string API_VERSION = '4.3' +const i64 MAX_QUERY_SIZE = 500 +//================================================= + +//----------------------------------------------------------------------------- +struct RunData{ + 1: i64 runId, // unique id of the run + 2: string runDate, // date of the run + 3: string name, // human readable name of the run + 4: i64 duration, // duration of the run; -1 if not finished + 5: i64 resultCount, // number of results in the run + 6: string runCmd, // the used check command + 7: optional bool can_delete // true if codeCheckerDBAccess::removeRunResults() + // is allowed on this run (see issue 151) +} +typedef list RunDataList + +//----------------------------------------------------------------------------- +struct ReportData{ + 1: string checkerId, // the qualified id of the checker that reported this + 2: string bugHash, // This is unique id of the concrete report. + 3: string checkedFile, // this is a filepath + 4: string checkerMsg, // description of the bug report + 5: i64 reportId, // id of the report in the current run in the db + 6: bool suppressed, // true if the bug is suppressed + 7: i64 fileId, // unique id of the file the report refers to + 8: shared.BugPathEvent lastBugPosition // This contains the range and message of the last item in the symbolic + // execution step list. + 9: shared.Severity severity // checker severity + 10: optional string suppressComment // suppress commment if report is suppressed +} +typedef list ReportDataList + +//----------------------------------------------------------------------------- +/** + * Members of this struct are interpreted in "AND" relation with each other. + * So they need to match a single report at the same time. + */ +struct ReportFilter{ + 1: optional string filepath, // In the filters a single wildcard can be be used: * + 2: optional string checkerMsg, + 3: optional shared.Severity severity, + 4: optional string checkerId, // should filter in the fully qualified checker id name such as alpha.core. + // the analyzed system. Projects can optionally use this concept. + 5: optional bool suppressed = false // if the bug state is suppressed +} + +/** + * If there is a list of ReportFilter, there is an OR relation between the list members. + */ +typedef list ReportFilterList + +//----------------------------------------------------------------------------- +struct ReportDetails{ + 1: shared.BugPathEvents pathEvents, + 2: shared.BugPath executionPath +} + +//----------------------------------------------------------------------------- +struct ReportFileData{ + 1: i64 reportFileId, + 2: string reportFileContent +} + +//----------------------------------------------------------------------------- +// default sorting of the results +enum SortType { + FILENAME, + CHECKER_NAME, + SEVERITY +} + +//----------------------------------------------------------------------------- +struct SourceFileData{ + 1: i64 fileId, + 2: string filePath, + 3: optional string fileContent +} + +//----------------------------------------------------------------------------- +enum Order { + ASC, + DESC +} + +//----------------------------------------------------------------------------- +struct SortMode{ + 1: SortType type, + 2: Order ord +} + +//----------------------------------------------------------------------------- +struct ReportDataTypeCount{ + 1: string checkerId, + 2: shared.Severity severity, + 3: i64 count +} +typedef list ReportDataTypeCountList + +//----------------------------------------------------------------------------- +struct SkipPathData{ + 1: string path, + 2: string comment +} +typedef list SkipPathDataList + +//----------------------------------------------------------------------------- +// diff result types +enum DiffType { + NEW, + RESOLVED, + UNRESOLVED +} + +//----------------------------------------------------------------------------- +service codeCheckerDBAccess { + + // get the run Ids and dates from the database to select one run + RunDataList getRunData() + throws (1: shared.RequestFailed requestError), + + ReportData getReport( + 1: i64 reportId) + throws (1: shared.RequestFailed requestError), + + // get the results for one runId + ReportDataList getRunResults( + 1: i64 runId, + 2: i64 limit, + 3: i64 offset, + 4: list sortType, + 5: ReportFilterList reportFilters) + throws (1: shared.RequestFailed requestError), + + // count all the results for one run + i64 getRunResultCount( + 1: i64 runId, + 2: ReportFilterList reportFilters) + throws (1: shared.RequestFailed requestError), + + // gives back the all marked region and message for a report + ReportDetails getReportDetails( + 1: i64 reportId) + throws (1: shared.RequestFailed requestError), + + // get file informations if fileContent is true the content of the source file + // will be also returned + SourceFileData getSourceFileData( + 1: i64 fileId, + 2: bool fileContent) + throws (1: shared.RequestFailed requestError), + + // get the file id from the database for a filepath, returns -1 if not found + i64 getFileId(1: i64 runId, + 2: string path) + throws (1: shared.RequestFailed requestError), + + // suppress the bug + bool suppressBug(1: list runIds, + 2: i64 reportId, + 3: string comment) + throws (1: shared.RequestFailed requestError), + + // unsuppress the bug + bool unSuppressBug(1: list runIds, + 2: i64 reportId) + throws (1: shared.RequestFailed requestError), + + // get the md documentation for a checker + string getCheckerDoc(1: string checkerId) + throws (1: shared.RequestFailed requestError), + + // compare the results of two runs + ReportDataList getNewResults(1: i64 base_run_id, + 2: i64 new_run_id, + 3: i64 limit, + 4: i64 offset, + 5: list sortType, + 6: ReportFilterList reportFilters ) + throws (1: shared.RequestFailed requestError), + + ReportDataList getResolvedResults(1: i64 base_run_id, + 2: i64 new_run_id, + 3: i64 limit, + 4: i64 offset, + 5: list sortType, + 6: ReportFilterList reportFilters ) + throws (1: shared.RequestFailed requestError), + + ReportDataList getUnresolvedResults(1: i64 base_run_id, + 2: i64 new_run_id, + 3: i64 limit, + 4: i64 offset, + 5: list sortType, + 6: ReportFilterList reportFilters ) + throws (1: shared.RequestFailed requestError), + + // get the checker configuration values + shared.CheckerConfigList getCheckerConfigs(1: i64 runId) + throws (1: shared.RequestFailed requestError), + + // get the skip list of paths + SkipPathDataList getSkipPaths(1: i64 runId) + throws (1: shared.RequestFailed requestError), + + // gives back the build Actions that generate the given report. + // multiple build actions can belong to a report in a header. + list getBuildActions(1: i64 reportId) + throws (1: shared.RequestFailed requestError), + + // get all the results for one runId + // count all results for a checker + ReportDataTypeCountList getRunResultTypes(1: i64 runId, + 2: ReportFilterList reportFilters) + throws (1: shared.RequestFailed requestError), + + // returns the database access handler api version + string getAPIVersion(); + + // remove bug results from the database + bool removeRunResults(1: list runIds) + throws (1: shared.RequestFailed requestError), + + // get the suppress file path set by the command line + // returns empty string if not set + string getSuppressFile() + throws (1: shared.RequestFailed requestError), + + // count the diff results + i64 getDiffResultCount(1: i64 base_run_id, + 2: i64 new_run_id, + 3: DiffType diff_type, + 4: ReportFilterList reportFilters) + throws (1: shared.RequestFailed requestError), + + // count all the diff results for each checker + ReportDataTypeCountList getDiffResultTypes(1: i64 base_run_id, + 2: i64 new_run_id, + 3: DiffType diff_type, + 4: ReportFilterList reportFilters) + throws (1: shared.RequestFailed requestError) + +} Index: tools/codechecker/thrift_api/shared.thrift =================================================================== --- /dev/null +++ tools/codechecker/thrift_api/shared.thrift @@ -0,0 +1,59 @@ +// -*- coding: utf-8 -*- +// The LLVM Compiler Infrastructure +// +// This file is distributed under the University of Illinois Open Source +// License. See LICENSE.TXT for details. + +//----------------------------------------------------------------------------- +struct BugPathEvent { + 1: i64 startLine, + 2: i64 startCol, + 3: i64 endLine, + 4: i64 endCol, + 5: string msg, + 6: i64 fileId + 7: string filePath +} +typedef list BugPathEvents + +//----------------------------------------------------------------------------- +struct BugPathPos { + 1: i64 startLine, + 2: i64 startCol, + 3: i64 endLine, + 4: i64 endCol, + 5: i64 fileId + 6: string filePath +} +typedef list BugPath + +//----------------------------------------------------------------------------- +struct ConfigValue { + 1: string checker_name, + 2: string attribute, + 3: string value +} +typedef list CheckerConfigList + +//----------------------------------------------------------------------------- +enum Severity{ + UNSPECIFIED = 0, + STYLE = 10, + LOW = 20, + MEDIUM = 30, + HIGH = 40, + CRITICAL = 50 +} + +//----------------------------------------------------------------------------- +enum ErrorCode{ + DATABASE, + IOERROR, + GENERAL +} + +//----------------------------------------------------------------------------- +exception RequestFailed { + 1: ErrorCode error_code, + 2: string message +} Index: tools/codechecker/thrift_api/thrift_api.md =================================================================== --- /dev/null +++ tools/codechecker/thrift_api/thrift_api.md @@ -0,0 +1,9 @@ + +#Thrift APIs +These APIs should be used by the clients using the database to store or to get the results. Any new client should only interact with the database through these APIs. + +## Report viewer server API +The report viewer server API should be used by any client to view the check results. + +## Report storage server API +The report storage server API is used internally in the package during runtime to store the results to the database.