[clang-tidy] Parallelize clang-tidy-diff.py

[zinovy.nis]

Summary

This patch has 2 rationales:

• large patches lead to long command lines and often cause max command line length restrictions imposed by OS;
• clang-tidy runs are independent and can be done in parallel, the same as done for run-clang-tidy.

[zinovy.nis]

Fixed minor typos.

[zinovy.nis]

Gentle ping.

[MyDeveloperDay]

Just a question.. If clang tidy is running with -fix in parallel, what stops each clang-tidy invocation altering a common header at the same time?

[zinovy.nis]

In D57662#1392509, @MyDeveloperDay wrote:

Just a question.. If clang tidy is running with -fix in parallel, what stops each clang-tidy invocation altering a common header at the same time?

You are right. May be it worth disabling -fix for j != 1.

[MyDeveloperDay]

You are right. May be it worth disabling -fix for j != 1.

I only say this because I think I might have seen it happen when I was running run-clang-tidy.py over a large code base with a fairly aggressive check/fixit, but frankly I was too new to LLVM to know it wasn't something I might have done wrong.

I saw my '[[nodiscard]]' checker splicing multiple [[nodiscard]]s onto the same line, I suspect there is a time of check/time of use issue for the locations, sometimes the positions where wrong by the time the Fixit came around. I guess because i'd used that script without specifiy a -j1 it just used the default which is -j0 which I guess goes as parallel as it can.

Its only when you raised this review for the diff check that I wondered if that could be the cause.

[zinovy.nis]

I only say this because I think I might have seen it happen when I was running run-clang-tidy.py over a large code base with a fairly aggressive check/fixit, but frankly I was too new to LLVM to know it wasn't something I might have done wrong.

A bit strange as run-clang-tidy.py explicitly collects fixes in separate files and after all merges them into a single fix in a thread/process-safe manner. Not like my current patch.

[zinovy.nis]

-j and -export-fixes were made mutually exclusive.

[zinovy.nis]

Any comments?

[JonasToth]

In D57662#1392509, @MyDeveloperDay wrote:

Just a question.. If clang tidy is running with -fix in parallel, what stops each clang-tidy invocation altering a common header at the same time?

clang-apply-replacements does it. The fixes are collected and then deduplicated and applied. If there are conflicts between the fixes at this stage they will be signaled.

[JonasToth]

clang-tidy/tool/clang-tidy-diff.py

53

if timeout is not None is more readable.

60

the command could be utf-8, too not?
If for example file-names use special characters they should be decoded as well. This is probably not done properly in the run-clang-tidy.py script, but can be adressed separatly.

78

why daemonize? Who kills that thread in the end? I think it's not the best choice.

162

I would prefer going the clang-apply-replacements route here as well.
It feels weird if there is an inconsistency between these parallel runners.

[zinovy.nis]

clang-tidy/tool/clang-tidy-diff.py

78

The script can be interrupted with keyabord or other signal, so it need not wait for any worker thread.

[alexfh]

clang-tidy/tool/clang-tidy-diff.py

129

The "clang-tidy runs are independent" assumption is unfortunately not valid for our internal compilation database integration, so making -j0 the default will break it. Let's make it 1.

[zinovy.nis]

clang-tidy/tool/clang-tidy-diff.py

129

Do you mean you run it with -fix or like this?

[alexfh]

clang-tidy/tool/clang-tidy-diff.py

129

Nope, our CompilationDatabase implementation doesn't support concurrent use (regardless of whether it's the same process or different ones).

[zinovy.nis]

• -j is 1 by default;
• fixed minor remarks;

[alexfh]

A few more comments.

clang-tidy/tool/clang-tidy-diff.py

60

nit: let's wrap this to 80 characters.

164

What's wrong with -export-fixes and -j? Different clang-tidy processes could export the fixes to different files (e.g. the -export-fixes= value could be used as a prefix of the filename with a sequential or random suffix. WDYT?

195

"start_workers" would be a clearer name, IMO.

[zinovy.nis]

clang-tidy/tool/clang-tidy-diff.py

164

I can copy/paste (shame on me!) a code from run-clang-tidy.py that does just what we need.

[zinovy.nis]

• honest '--export-fixes': multiple yamls are merged into the resulting one.
• minor cosmetic fixes.

[zinovy.nis]

Gentle ping.

[zinovy.nis]

One more gentle ping.

[alexfh]

LG with a nit

clang-tidy/tool/clang-tidy-diff.py

90

nit: Spaces around operators, please

[derek]

@zinovy.nis @alexfh @JonasToth @MyDeveloperDay

Hi folks, please correct me if I'm wrong but it appears that an effect of this change is that this script will no longer exit non-zero if clang-tidy discovers any errors, which was the previous functionality with sys.exit(subprocess.call(' '.join(command), shell=True)). As a result, when we upgraded a project to LLVM 9 our CI began having false-positives, as we relied on the exit code of this script to indicate success/failure. Would it be possible to restore that functionality? I'd be happy to provide a patch.

[zinovy.nis]

In D57662#1797607, @derek wrote:

@zinovy.nis @alexfh @JonasToth @MyDeveloperDay

I'd be happy to provide a patch.

It would be nice, thanks!

[JonasToth]

In D57662#1797607, @derek wrote:

@zinovy.nis @alexfh @JonasToth @MyDeveloperDay

Hi folks, please correct me if I'm wrong but it appears that an effect of this change is that this script will no longer exit non-zero if clang-tidy discovers any errors, which was the previous functionality with sys.exit(subprocess.call(' '.join(command), shell=True)). As a result, when we upgraded a project to LLVM 9 our CI began having false-positives, as we relied on the exit code of this script to indicate success/failure. Would it be possible to restore that functionality? I'd be happy to provide a patch.

Yeah, that would be good to restore. run-clang-tidy.py is parallelized and does fail properly (95% sure about that xD), so in doubt you can check that script as well.

clang-tidy/tool/clang-tidy-diff.py

Example usage for git/svn users:
 
   git diff -U0 HEAD^ | clang-tidy-diff.py -p1
   svn diff --diff-cmd=diff -x-U0 | \
       clang-tidy-diff.py -fix -checks=-*,modernize-use-override
 
 """
 
 import argparse
+import glob
 import json
+import multiprocessing
+import os
 import re
+import shutil
 import subprocess
 import sys
+import tempfile
+import threading
+import traceback
+import yaml
+
+is_py2 = sys.version[0] == '2'
+
+if is_py2:
+    import Queue as queue
+else:
+    import queue as queue
+
+
+def run_tidy(task_queue, lock, timeout):
+  watchdog = None
+  while True:
+    command = task_queue.get()
+    try:
+      proc = subprocess.Popen(command,
+                              stdout=subprocess.PIPE,
+                              stderr=subprocess.PIPE)
+
+      if timeout is not None:
+        watchdog = threading.Timer(timeout, proc.kill)
+        watchdog.start()
+
+      stdout, stderr = proc.communicate()
+
+      with lock:
+        sys.stdout.write(stdout.decode('utf-8') + '\n')
+        if stderr:
+          sys.stderr.write(stderr.decode('utf-8') + '\n')
+    except Exception as e:
+      with lock:
+        sys.stderr.write('Failed: ' + str(e) + ': '.join(command) + '\n')
+    finally:
+      with lock:
+        if (not timeout is None) and (not watchdog is None):
+          if not watchdog.is_alive():
+              sys.stderr.write('Terminated by timeout: ' +
+                               ' '.join(command) + '\n')
+          watchdog.cancel()
+      task_queue.task_done()
+
+
+def start_workers(max_tasks, tidy_caller, task_queue, lock, timeout):
+  for _ in range(max_tasks):
+    t = threading.Thread(target=tidy_caller, args=(task_queue, lock, timeout))
+    t.daemon = True
+    t.start()
+
+def merge_replacement_files(tmpdir, mergefile):
+  """Merge all replacement files in a directory into a single file"""
+  # The fixes suggested by clang-tidy >= 4.0.0 are given under
+  # the top level key 'Diagnostics' in the output yaml files
+  mergekey="Diagnostics"
+  merged=[]
+  for replacefile in glob.iglob(os.path.join(tmpdir, '*.yaml')):
+    content = yaml.safe_load(open(replacefile, 'r'))
+    if not content:
+      continue # Skip empty files.
+    merged.extend(content.get(mergekey, []))
+
+  if merged:
+    # MainSourceFile: The key is required by the definition inside
+    # include/clang/Tooling/ReplacementsYaml.h, but the value
+    # is actually never used inside clang-apply-replacements,
+    # so we set it to '' here.
+    output = { 'MainSourceFile': '', mergekey: merged }
+    with open(mergefile, 'w') as out:
+      yaml.safe_dump(output, out)
+  else:
+    # Empty the file:
+    open(mergefile, 'w').close()
 
 
 def main():
   parser = argparse.ArgumentParser(description=
                                    'Run clang-tidy against changed files, and '
                                    'output diagnostics only for modified '
                                    'lines.')
   parser.add_argument('-clang-tidy-binary', metavar='PATH',
                       default='clang-tidy',
                       help='path to clang-tidy binary')
   parser.add_argument('-p', metavar='NUM', default=0,
                       help='strip the smallest prefix containing P slashes')
   parser.add_argument('-regex', metavar='PATTERN', default=None,
                       help='custom pattern selecting file paths to check '
                       '(case sensitive, overrides -iregex)')
   parser.add_argument('-iregex', metavar='PATTERN', default=
                       r'.*\.(cpp|cc|c\+\+|cxx|c|cl|h|hpp|m|mm|inc)',
                       help='custom pattern selecting file paths to check '
                       '(case insensitive, overridden by -regex)')
 
+  parser.add_argument('-j', type=int, default=1,
+                      help='number of tidy instances to be run in parallel.')
+  parser.add_argument('-timeout', type=int, default=None,
+                      help='timeout per each file in seconds.')
   parser.add_argument('-fix', action='store_true', default=False,
                       help='apply suggested fixes')
   parser.add_argument('-checks',
                       help='checks filter, when not specified, use clang-tidy '
                       'default',
                       default='')
   parser.add_argument('-path', dest='build_path',
                       help='Path used to read a compile command database.')
def main():
   clang_tidy_args = []
   argv = sys.argv[1:]
   if '--' in argv:
     clang_tidy_args.extend(argv[argv.index('--'):])
     argv = argv[:argv.index('--')]
 
   args = parser.parse_args(argv)
 
   # Extract changed lines for each file.
   filename = None
   lines_by_file = {}
   for line in sys.stdin:
     match = re.search('^\+\+\+\ \"?(.*?/){%s}([^ \t\n\"]*)' % args.p, line)
     if match:
       filename = match.group(2)
-    if filename == None:
+    if filename is None:
       continue
 
     if args.regex is not None:
       if not re.match('^%s$' % args.regex, filename):
         continue
     else:
       if not re.match('^%s$' % args.iregex, filename, re.IGNORECASE):
         continue
 
     match = re.search('^@@.*\+(\d+)(,(\d+))?', line)
     if match:
       start_line = int(match.group(1))
       line_count = 1
       if match.group(3):
         line_count = int(match.group(3))
       if line_count == 0:
         continue
-      end_line = start_line + line_count - 1;
+      end_line = start_line + line_count - 1
       lines_by_file.setdefault(filename, []).append([start_line, end_line])
 
-  if len(lines_by_file) == 0:
+  if not any(lines_by_file):
     print("No relevant changes found.")
     sys.exit(0)
 
-  line_filter_json = json.dumps(
-    [{"name" : name, "lines" : lines_by_file[name]} for name in lines_by_file],
-    separators = (',', ':'))
+  max_task_count = args.j
+  if max_task_count == 0:
+      max_task_count = multiprocessing.cpu_count()
+  max_task_count = min(len(lines_by_file), max_task_count)
 
-  quote = "";
-  if sys.platform == 'win32':
-    line_filter_json=re.sub(r'"', r'"""', line_filter_json)
-  else:
-    quote = "'";
+  tmpdir = None
+  if args.export_fixes:
+    tmpdir = tempfile.mkdtemp()
 
-  # Run clang-tidy on files containing changes.
-  command = [args.clang_tidy_binary]
-  command.append('-line-filter=' + quote + line_filter_json + quote)
+  # Tasks for clang-tidy.
+  task_queue = queue.Queue(max_task_count)
+  # A lock for console output.
+  lock = threading.Lock()
+
+  # Run a pool of clang-tidy workers.
+  start_workers(max_task_count, run_tidy, task_queue, lock, args.timeout)
+
+  quote = ""
+  if sys.platform != 'win32':
+    quote = "'"
+
+  # Form the common args list.
+  common_clang_tidy_args = []
   if args.fix:
-    command.append('-fix')
-  if args.export_fixes:
-    command.append('-export-fixes=' + args.export_fixes)
+    common_clang_tidy_args.append('-fix')
   if args.checks != '':
-    command.append('-checks=' + quote + args.checks + quote)
+    common_clang_tidy_args.append('-checks=' + quote + args.checks + quote)
   if args.quiet:
-    command.append('-quiet')
+    common_clang_tidy_args.append('-quiet')
   if args.build_path is not None:
-    command.append('-p=%s' % args.build_path)
-  command.extend(lines_by_file.keys())
+    common_clang_tidy_args.append('-p=%s' % args.build_path)
   for arg in args.extra_arg:
-      command.append('-extra-arg=%s' % arg)
+    common_clang_tidy_args.append('-extra-arg=%s' % arg)
   for arg in args.extra_arg_before:
-      command.append('-extra-arg-before=%s' % arg)
+    common_clang_tidy_args.append('-extra-arg-before=%s' % arg)
+
+  for name in lines_by_file:
+    line_filter_json = json.dumps(
+      [{"name": name, "lines": lines_by_file[name]}],
+      separators=(',', ':'))
+
+    # Run clang-tidy on files containing changes.
+    command = [args.clang_tidy_binary]
+    command.append('-line-filter=' + quote + line_filter_json + quote)
+    if args.export_fixes:
+      # Get a temporary file. We immediately close the handle so clang-tidy can
+      # overwrite it.
+      (handle, tmp_name) = tempfile.mkstemp(suffix='.yaml', dir=tmpdir)
+      os.close(handle)
+      command.append('-export-fixes=' + tmp_name)
+    command.extend(common_clang_tidy_args)
+    command.append(name)
     command.extend(clang_tidy_args)
 
-  sys.exit(subprocess.call(' '.join(command), shell=True))
+    task_queue.put(command)
+
+  # Wait for all threads to be done.
+  task_queue.join()
+
+  if args.export_fixes:
+    print('Writing fixes to ' + args.export_fixes + ' ...')
+    try:
+      merge_replacement_files(tmpdir, args.export_fixes)
+    except:
+      sys.stderr.write('Error exporting fixes.\n')
+      traceback.print_exc()
+
+  if tmpdir:
+    shutil.rmtree(tmpdir)
+
 
 if __name__ == '__main__':
   main()