[clang-tidy] Add a little python script that can run clang-tidy and apply fixes over an entire codebase.

[bkramer]

Summary

Ever wanted to fix all the header guards in clang? Now it's easy.

Make sure clang-tidy is in $PATH and a compilation database is available.
$ ./run-clang-tidy.py -checks=llvm-header-guard -fix
... get coffee (or more CPU cores) ...
$ svn diff

Some may argue that this is just a glorified xargs -P, but it does a bit more ;)

[alexfh]

clang-tidy/tool/run-clang-tidy.py

34

I'd use a more specific term like "Compilation database".

109

Why not import it in the beginning?

121

I'm still a bit concerned about reimplementing xargs in Python. Especially if you have to handle ctrl-c manually.

[bkramer]

clang-tidy/tool/run-clang-tidy.py

109

Mostly because it's heavyweight and we only need this tiny bit of information. Thinking more about it I could also move it to the top.

121

In fact I had a version lying around that just calls xargs, but I don't know how to make it generate a random file name for each clang-tidy invocation. We use it to write out the fixits yaml.

[bkramer]

Move import to the top.
Make it more obvious that we're refering to the compilation db.

[klimek]

clang-tidy/tool/run-clang-tidy.py

12

FIXME

14

Is that for the \ at the end of the run-clang-tidy invocation?

64–67

The second branch might need a comment.

72–73

Unsafe how? Also, why is it ok for us?

118

Call that file_name_re or something? Then you can remove the comment on bigre.search...

128–131

Since we have to break anyway, call it invocation. Otherwise I'll always read it as invitation. Which is kinda funny, but in a sad way.

145–146

Uuuh. Don't we have some way to pool() multiple processes?

148–154

This reads like one of Shakespeare's tragicomedies. Just less entertaining. Oh, Python.

157–163

Pull into a function.

159–160

Indent.

[bkramer]

• Update & add comments.
• Pulled apply_fixes into a function.

[bkramer]

clang-tidy/tool/run-clang-tidy.py

145–146

There is multiprocessing.Pool which I couldn't get to work properly (it's part of the thread-with-processes emulation and adds all kinds of weird behavior to child processes). Then there's select which doesn't work on windows. Just sleeping seemed like the best option.

[bkramer]

Remove experiment that broke everything, sorry.

[bkramer]

Replace Popen + sleep hack with thread hack.

[klimek]

clang-tidy/tool/run-clang-tidy.py

77

Use mkstemp!

[bkramer]

Use mkstemp!

[klimek]

LG

Closed by commit rL217368 (authored by d0k).

[alexfh]

clang-tidy/tool/run-clang-tidy.py

121

Actually, if we want to support Windows properly, we can't rely on xargs and bash. So it may be a reasonable sacrifice to the gods of portability.

clang-tidy/tool/run-clang-tidy.py

+#!/usr/bin/env python
+#
+#===- run-clang-tidy.py - Parallel clang-tidy runner ---------*- python -*--===#
+#
+#                     The LLVM Compiler Infrastructure
+#
+# This file is distributed under the University of Illinois Open Source
+# License. See LICENSE.TXT for details.
+#
+#===------------------------------------------------------------------------===#
+# FIXME: Integrate with clang-tidy-diff.py
+
+"""
+Parallel clang-tidy runner
+==========================
+
+Runs clang-tidy over all files in a compilation database. Requires clang-tidy
+and clang-apply-replacements in $PATH.
+
+Example invocations.
+- Run clang-tidy on all files in the current working directory with a default
+  set of checks and show warnings in the cpp files and all project headers.
+    run-clang-tidy.py $PWD
+
+- Fix all header guards.
+    run-clang-tidy.py -fix -checks=-*,llvm-header-guard
+
+- Fix all header guards included from clang-tidy and header guards
+  for clang-tidy headers.
+    run-clang-tidy.py -fix -checks=-*,llvm-header-guard extra/clang-tidy \
+                      -header-filter=extra/clang-tidy
+
+Compilation database setup:
+http://clang.llvm.org/docs/HowToSetupToolingForLLVM.html
+"""
+
+import argparse
+import json
+import multiprocessing
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+
+
+def find_compilation_database(path):
+  """Adjusts the directory until a compilation database is found."""
+  result = './'
+  while not os.path.isfile(os.path.join(result, path)):
+    if os.path.realpath(result) == '/':
+      print 'Error: could not find compilation database.'
+      sys.exit(1)
+    result += '../'
+  return os.path.realpath(result)
+
+
+def get_tidy_invocation(f, checks, tmpdir, build_path, header_filter):
+  """Gets a command line for clang-tidy."""
+  start = ['clang-tidy']
+  if header_filter is not None:
+    start.append('-header-filter=' + header_filter)
+  else:
+    # Show warnings in all in-project headers by default.
+    start.append('-header-filter=^' + build_path + '/.*')
+  if checks:
+    start.append('-checks=-*,' + checks)
+  if tmpdir is not None:
+    start.append('-export-fixes')
+    # mktemp is potentially unsafe but the only reasonable way to get a
+    # temporary file name. Name collisions are extremely rare and in the worst
+    # case we'll lose edits for a file.
+    start.append(tempfile.mktemp(suffix='.yaml', dir=tmpdir))
+  start.append('-p=' + build_path)
+  start.append(f)
+  return start
+
+
+def apply_fixes(tmpdir, format):
+  invocation = ['clang-apply-replacements']
+  if format:
+    invocation.append('-format')
+  invocation.append(tmpdir)
+  subprocess.call(invocation)
+  shutil.rmtree(tmpdir)
+
+
+def main():
+  parser = argparse.ArgumentParser(description='Runs clang-tidy over all files '
+                                   'in a compilation database. Requires '
+                                   'clang-tidy and clang-apply-replacements in '
+                                   '$PATH.')
+  parser.add_argument('-checks', default=None,
+                      help='checks filter, when not specified, use clang-tidy '
+                      'default')
+  parser.add_argument('-header-filter', default=None,
+                      help='regular expression matching the names of the '
+                      'headers to output diagnostics from. Diagnostics from '
+                      'the main file of each translation unit are always '
+                      'displayed.')
+  parser.add_argument('-j', type=int, default=0,
+                      help='number of tidy instances to be run in parallel.')
+  parser.add_argument('files', nargs='*', default=['.*'],
+                      help='files to be processed (regex on path)')
+  parser.add_argument('-fix', action='store_true', help='apply fix-its')
+  parser.add_argument('-format', action='store_true', help='Reformat code '
+                      'after applying fixes')
+  args = parser.parse_args()
+
+  # Find our database.
+  db_path = 'compile_commands.json'
+  build_path = find_compilation_database(db_path)
+
+  # Load the database and extract all files.
+  database = json.load(open(os.path.join(build_path, db_path)))
+  files = [entry['file'] for entry in database]
+
+  max_task = args.j
+  if max_task == 0:
+    max_task = multiprocessing.cpu_count()
+
+  tmpdir = None
+  if args.fix:
+    tmpdir = tempfile.mkdtemp()
+
+  # Build up a big regexy filter from all command line arguments.
+  file_name_re = re.compile('(' + ')|('.join(args.files) + ')')
+
+  processes = []
+  retval = 0
+  try:
+    while True:
+      # Spin up some clang-tidy instances.
+      while files and len(processes) < max_task:
+        name = files.pop()
+        if file_name_re.search(name):
+          invocation = get_tidy_invocation(name, args.checks, tmpdir,
+                                           build_path, args.header_filter)
+          print ' '.join(invocation)
+          processes.append(subprocess.Popen(invocation))
+
+      # Remove finished processes from the list. Remember the return code
+      # for later.
+      for p in processes:
+        if p.poll() is not None:
+          if p.returncode != 0:
+            retval = p.returncode
+          processes.remove(p)
+
+      # Done yet?
+      if not processes and not files:
+        break
+
+      # os.wait() would be nicer, but it confuses Popen.
+      time.sleep(0.05)
+
+  except KeyboardInterrupt:
+    # This is a sad hack. Unfortunately subprocess goes
+    # bonkers with ctrl-c and we start forking merrily.
+    print '\nCtrl-C detected, goodbye.'
+    if args.fix:
+      shutil.rmtree(tmpdir)
+    os.kill(0, 9)
+
+  if args.fix:
+    print 'Applying fixes ...'
+    apply_fixes(tmpdir, args.format)
+
+  sys.exit(retval)
+
+if __name__ == '__main__':
+  main()