[clang-tidy] Add a useful note about -std=c++11-or-later

[steakhal]

Summary

I and @whisperity spent some time debugging a LIT test case using the
-std=c++11-or-later check_clang_tidy.py flag when the test had
fixits.

It turns out if the test wants to report a diagnostic into a header
file AND into the test file as well, one needs to first copy the header
somewhere under the build directory.
It needs to be copied since clang-tidy sorts the reports into
alphabetical order, thus to have a deterministic order relative to the
diagnostic in the header AND the diagnostic in the test cpp file.

There is more to this story.

The -std=c++11-or-later turns out executes the test with multiple
-std=XX version substitution and each execution will also have the
-fix tidy parameter. This means that the freshly copied header file I
stated in the previous paragraph gets fixed up and the very next tidy
execution will fail miserably.

Following @whisperity's advice, I'm leaving a reminder about such
shared state in the related doc comment.

[martong]

clang-tools-extra/test/clang-tidy/check_clang_tidy.py

31

typo

[steakhal]

fix typo withing -> within

[LegalizeAdulthood]

I thought there wasn't any support for validating fixits applied to header files?

[whisperity]

In D125771#3519606, @LegalizeAdulthood wrote:

I thought there wasn't any support for validating fixits applied to header files?

It is not specifically about the fixits, but diagnostics as a whole. It was not clear that if you say -std=c++11-or-later it will actually run multiple executions without giving you the ability to observe the contents of the files.

Technically, you can always match a diagnostic that is positioned into a header from the TU that is producing that diagnostic, because // CHECK-MESSAGES: does a regular expression-like matching. For example, misc-unused-parameters does the following:

#include "header.h"
// CHECK-MESSAGES: header.h:1:38: warning

And because LIT can execute arbitrary commands, you can always go about your way running FileCheck and everything else manually.

But it is good to be reminded early on that -or-later will result in execution of potentially changed code outside of your (test file's) control. Saves the debugging time which we had to endure!

[njames93]

Tbh the whole testing infrastructure for clang-tidy is a mess. When I have wanted to verify fixes and diagnostics for header files, I find manually invoking clang-tidy is better than using the check_clang_tidy script.
I don't have the time right now, but a verify mode like with clang is something that we should move to in the future.

[LegalizeAdulthood]

In D125771#3525252, @njames93 wrote:

Tbh the whole testing infrastructure for clang-tidy is a mess. When I have wanted to verify fixes and diagnostics for header files, I find manually invoking clang-tidy is better than using the check_clang_tidy script.
I don't have the time right now, but a verify mode like with clang is something that we should move to in the future.

I had a change, literally years ago at this point, that provided support for verifying header files, but it got bogged down in review nits and never landed.

[LegalizeAdulthood]

clang-tools-extra/test/clang-tidy/check_clang_tidy.py

22

These aren't the only two valid options for this argument.... Valid values are:

• c++98-or-later
• c++11-or-later
• c++14-or-later
• c++17-or-later
• c++20-or-later

and of course all the specific standard versions:

• c++98
• c++11
• c++14
• c++17
• c++20

30

Probably need to explain the -or-later argument option more generally here and not just for C++11

[steakhal]

Generalized the flag help text to all supported standard versions.

[LegalizeAdulthood]

clang-tools-extra/test/clang-tidy/check_clang_tidy.py

31

It's only the -or-later variants that cause multiple runs, so we should explain this more explicitly.

[steakhal]

Make the note more precise.

clang-tools-extra/test/clang-tidy/check_clang_tidy.py

31

Oh, true!

[steakhal]

Thanks for the review, sorry about the mistakes I made on this trivial change.

clang-tools-extra/test/clang-tidy/check_clang_tidy.py

 
 This script runs clang-tidy in fix mode and verify fixes, messages or both.
 
 Usage:
   check_clang_tidy.py [-resource-dir=<resource-dir>] \
     [-assume-filename=<file-with-source-extension>] \
     [-check-suffix=<comma-separated-file-check-suffixes>] \
     [-check-suffixes=<comma-separated-file-check-suffixes>] \
+    [-std=(c++11-or-later|c++17)] \
     <source-file> <check-name> <temp-file> \
     -- [optional clang-tidy arguments]
 
 Example:
   // RUN: %check_clang_tidy %s llvm-include-order %t -- -- -isystem %S/Inputs
+
+Notes:
+  -std=c++11-or-later:
+    This flag will cause multiple runs within the same check_clang_tidy
+    execution. Make sure you don't have shared state across these runs.
 """
 
 import argparse
 import os
 import re
 import subprocess
 import sys