[clang-tidy] Customize FileCheck prefix in check_clang-tidy.py

[zinovy.nis]

Summary

The patch introduces a new command line option -check_suffix for check_clang_tidy.py to allow multiple %check_clang_tidy% in tests.

Sample:

// RUN: %check_clang_tidy -check_suffix=-FLAG_1 %s misc-unused-using-decls %t -- -- <option set 1>
// RUN: %check_clang_tidy -check_suffix=-FLAG_2 %s misc-unused-using-decls %t -- -- <option set 2> 
...
+// CHECK-MESSAGES-FLAG_1: :[[@LINE-4]]:10: warning: using decl 'B' is unused [misc-unused-using-decls]
+// CHECK-MESSAGES-FLAG_2: :[[@LINE-7]]:10: warning: using decl 'A' is unused [misc-unused-using-decls]
+// CHECK-FIXES-FLAG_1-NOT: using a::A;$
+// CHECK-FIXES-FLAG_2-NOT: using a::B;$

[alexfh]

Thank you for the contribution!
Please update the documentation accordingly (http://clang.llvm.org/extra/clang-tidy/#testing-checks).

// RUN: %check_clang_tidy %s misc-unused-using-decls %t -check_suffix=-FLAG_1-- <option set 1>

I don't know whether it makes sense to endorse (or even allow) the use of underscore in the check suffix. The mix of underscores and dashes looks ugly and is prone to errors.

test/clang-tidy/check_clang_tidy.cpp

21

Fix "No newline at end of file"

test/clang-tidy/check_clang_tidy.py

45

Please use dash as a separator as in other arguments.

77

Maybe the script should add a dash when check_suffix is not empty, so that one could use -check-suffix=FLAG instead of -check-suffix=-FLAG?

103

Looks like debug output. Remove?

[zinovy.nis]

• Fixed issues pointed by Alexander.

[zinovy.nis]

• Updated docs.

[zinovy.nis]

• Removed exec attribute on file.

[zinovy.nis]

Done.

[zinovy.nis]

test/clang-tidy/check_clang_tidy.py

77

OK.

[zinovy.nis]

• Minor cosmetic fixes.

[alexfh]

A few style-related comments.

docs/clang-tidy/index.rst

677

s/SUFFIX_NAME/SUFFIX-NAME/

Same below.

685

I'd use the `-check-suffix=...` format (with the equals sign) for consistency with the documentation.

test/clang-tidy/check_clang_tidy.py

21

`-check-suffix=<file-check-suffix>` and maybe the same for -assume-filename

[zinovy.nis]

• Applied the changes suggested by Alexander.

[alexfh]

LG

[lebedev.ri]

Did this intentionally omit the possibility to pass a group of prefixes, like FileCheck supports?
Usefulness of this feature is somewhat penalized by this.

[zinovy.nis]

In D45776#1251256, @lebedev.ri wrote:

Did this intentionally omit the possibility to pass a group of prefixes, like FileCheck supports?
Usefulness of this feature is somewhat penalized by this.

I did not fully realize what do you mean. Can you please provide a sample? I see only single -check-prefix=<string> in FileCheck.

[lebedev.ri]

In D45776#1253934, @zinovy.nis wrote:

In D45776#1251256, @lebedev.ri wrote:

Did this intentionally omit the possibility to pass a group of prefixes, like FileCheck supports?
Usefulness of this feature is somewhat penalized by this.

I did not fully realize what do you mean. Can you please provide a sample? I see only single -check-prefix=<string> in FileCheck.

https://llvm.org/docs/CommandGuide/FileCheck.html#cmdoption-check-prefixes

[lebedev.ri]

In D45776#1253934, @zinovy.nis wrote:

In D45776#1251256, @lebedev.ri wrote:

Did this intentionally omit the possibility to pass a group of prefixes, like FileCheck supports?
Usefulness of this feature is somewhat penalized by this.

I did not fully realize what do you mean. Can you please provide a sample? I see only single -check-prefix=<string> in FileCheck.

https://llvm.org/docs/CommandGuide/FileCheck.html#cmdoption-check-prefixes

[zinovy.nis]

In D45776#1253944, @lebedev.ri wrote:

Did this intentionally omit the possibility to pass a group of prefixes, like FileCheck supports?
Usefulness of this feature is somewhat penalized by this.

Done. Please review this https://reviews.llvm.org/D52971

docs/clang-tidy/index.rst

.. code-block:: c++
   // RUN: %check_clang_tidy %s google-readability-casting %t
 
   void f(int a) {
     int b = (int)a;
     // CHECK-MESSAGES: :[[@LINE-1]]:11: warning: redundant cast to the same type [google-readability-casting]
     // CHECK-FIXES: int b = a;
   }
 
+To check more than one scenario in the same test file use 
+``-check-suffix=SUFFIX_NAME`` on ``check_clang_tidy.py`` command line.
+With ``-check-suffix=SUFFIX_NAME`` you need to replace your ``CHECK-*`` 
+directives with ``CHECK-MESSAGES-SUFFIX-NAME`` and ``CHECK-FIXES-SUFFIX-NAME``.
+
+Here's an example:
+
+.. code-block:: c++
+
+   // RUN: %check_clang_tidy %s misc-unused-using-decls %t -check-suffix=USING-A -- -DUSING_A
+   // RUN: %check_clang_tidy %s misc-unused-using-decls %t -check-suffix=USING-B -- -DUSING_B
+   // RUN: %check_clang_tidy %s misc-unused-using-decls %t
+   ...
+   // CHECK-MESSAGES-USING-A: :[[@LINE-8]]:10: warning: using decl 'A' {{.*}}
+   // CHECK-MESSAGES-USING-B: :[[@LINE-7]]:10: warning: using decl 'B' {{.*}}
+   // CHECK-MESSAGES: :[[@LINE-6]]:10: warning: using decl 'C' {{.*}}
+   // CHECK-FIXES-USING-A-NOT: using a::A;$
+   // CHECK-FIXES-USING-B-NOT: using a::B;$
+   // CHECK-FIXES-NOT: using a::C;$
+
+
 There are many dark corners in the C++ language, and it may be difficult to make
 your check work perfectly in all cases, especially if it issues fix-it hints. The
 most frequent pitfalls are macros and templates:
 
 1. code written in a macro body/template definition may have a different meaning
    depending on the macro expansion/template instantiation;
 2. multiple macro expansions/template instantiations may result in the same code
    being inspected by the check multiple times (possibly, with different

test/clang-tidy/check_clang_tidy.cpp

+// RUN: %check_clang_tidy %s misc-unused-using-decls %t -check-suffix=USING-A -- -DUSING_A
+// RUN: %check_clang_tidy %s misc-unused-using-decls %t -check-suffix=USING-B -- -DUSING_B
+// RUN: %check_clang_tidy %s misc-unused-using-decls %t
+namespace a {class A {}; class B {}; class C {}; }
+namespace b {
+#if defined(USING_A)
+using a::A;
+#elif  defined(USING_B)
+using a::B;
+#else
+using a::C;
+#endif
+}
+namespace c {}
+// CHECK-MESSAGES-USING-A: :[[@LINE-8]]:10: warning: using decl 'A' {{.*}}
+// CHECK-MESSAGES-USING-B: :[[@LINE-7]]:10: warning: using decl 'B' {{.*}}
+// CHECK-MESSAGES: :[[@LINE-6]]:10: warning: using decl 'C' {{.*}}
+// CHECK-FIXES-USING-A-NOT: using a::A;$
+// CHECK-FIXES-USING-B-NOT: using a::B;$
+// CHECK-FIXES-NOT: using a::C;$

test/clang-tidy/check_clang_tidy.py

 ClangTidy Test Helper
 =====================
 
 This script runs clang-tidy in fix mode and verify fixes, messages or both.
 
 Usage:
   check_clang_tidy.py [-resource-dir <resource-dir>] \
     [-assume-filename <file-with-source-extension>] \
     <source-file> <check-name> <temp-file> \
     -- [optional clang-tidy arguments]
 
 Example:
   // RUN: %check_clang_tidy %s llvm-include-order %t -- -- -isystem %S/Inputs
 """
 
 import argparse
 import os
 import re
 import subprocess
 import sys
 
 
 def write_file(file_name, text):
   with open(file_name, 'w') as f:
     f.write(text)
     f.truncate()
 
 def main():
   parser = argparse.ArgumentParser()
   parser.add_argument('-expect-clang-tidy-error', action='store_true')
   parser.add_argument('-resource-dir')
   parser.add_argument('-assume-filename')
+  parser.add_argument('-check-suffix', default='')
   parser.add_argument('input_file_name')
   parser.add_argument('check_name')
   parser.add_argument('temp_file_name')
 
   args, extra_args = parser.parse_known_args()
 
   resource_dir = args.resource_dir
   assume_file_name = args.assume_filename
def main():
   if len(clang_tidy_extra_args) == 0:
     clang_tidy_extra_args = ['--']
     if extension in ['.cpp', '.hpp', '.mm']:
       clang_tidy_extra_args.append('--std=c++11')
     if extension in ['.m', '.mm']:
       clang_tidy_extra_args.extend(
           ['-fobjc-abi-version=2', '-fobjc-arc'])
 
+  if args.check_suffix and not re.match('^[A-Z0-9\-]+$', args.check_suffix):
+    sys.exit('Only A..Z, 0..9 and "-" are allowed in check suffix, but "%s" was given' % (args.check_suffix))
+
+  file_check_suffix = ('-' + args.check_suffix) if args.check_suffix else ''
+  check_fixes_prefix = 'CHECK-FIXES' + file_check_suffix
+  check_messages_prefix = 'CHECK-MESSAGES' + file_check_suffix
+
   # Tests should not rely on STL being available, and instead provide mock
   # implementations of relevant APIs.
   clang_tidy_extra_args.append('-nostdinc++')
 
   if resource_dir is not None:
     clang_tidy_extra_args.append('-resource-dir=%s' % resource_dir)
 
   with open(input_file_name, 'r') as input_file:
     input_text = input_file.read()
 
-  has_check_fixes = input_text.find('CHECK-FIXES') >= 0
-  has_check_messages = input_text.find('CHECK-MESSAGES') >= 0
+  has_check_fixes = check_fixes_prefix in input_text
+  has_check_messages = check_messages_prefix in input_text
 
   if not has_check_fixes and not has_check_messages:
-    sys.exit('Neither CHECK-FIXES nor CHECK-MESSAGES found in the input')
+    sys.exit('Neither %s nor %s found in the input' % (check_fixes_prefix, check_messages_prefix) )
 
   # Remove the contents of the CHECK lines to avoid CHECKs matching on
   # themselves.  We need to keep the comments to preserve line numbers while
   # avoiding empty lines which could potentially trigger formatting-related
   # checks.
-  cleaned_test = re.sub('// *CHECK-[A-Z-]*:[^\r\n]*', '//', input_text)
+  cleaned_test = re.sub('// *CHECK-[A-Z0-9\-]*:[^\r\n]*', '//', input_text)
 
   write_file(temp_file_name, cleaned_test)
 
   original_file_name = temp_file_name + ".orig"
   write_file(original_file_name, cleaned_test)
 
   args = ['clang-tidy', temp_file_name, '-fix', '--checks=-*,' + check_name] + \
         clang_tidy_extra_args
   if expect_clang_tidy_error:
     args.insert(0, 'not')
def main():
   print('------------------------------ Fixes -----------------------------\n' +
         diff_output.decode() +
         '\n------------------------------------------------------------------')
 
   if has_check_fixes:
     try:
       subprocess.check_output(
           ['FileCheck', '-input-file=' + temp_file_name, input_file_name,
-           '-check-prefix=CHECK-FIXES', '-strict-whitespace'],
+           '-check-prefix=' + check_fixes_prefix, '-strict-whitespace'],
           stderr=subprocess.STDOUT)
     except subprocess.CalledProcessError as e:
       print('FileCheck failed:\n' + e.output.decode())
       raise
 
   if has_check_messages:
     messages_file = temp_file_name + '.msg'
     write_file(messages_file, clang_tidy_output)
     try:
       subprocess.check_output(
           ['FileCheck', '-input-file=' + messages_file, input_file_name,
-           '-check-prefix=CHECK-MESSAGES',
+           '-check-prefix=' + check_messages_prefix,
            '-implicit-check-not={{warning|error}}:'],
           stderr=subprocess.STDOUT)
     except subprocess.CalledProcessError as e:
       print('FileCheck failed:\n' + e.output.decode())
       raise
 
 if __name__ == '__main__':
   main()