[clang-tidy] Fix checks filter with warnings-as-errors

[PiotrZSL]

Summary

Since commit 5d12b13b0b26bc58b02ee23c369da8b83240cceb, warnings are
internally classified as errors which skip the check filters.

One use-case is particularly affected: you cannot selectively disable
clang-analyzer-core checks, they are force-enabled because required by others.
So enabling warning as errors will show new (and unwanted) errors !

Co-authored-by: kiwixz <[e-mail]>

Fixes: #61969

[PiotrZSL]

Looks fine, and looks like it's working.

[carlosgalvezp]

I'm not sure I follow - if you enable warnings as error for all checks, then the expectation is that you indeed get an error if you violate one of the clang-analyzer checks. In what way is this not wanted?

If you could post a minimal self-contained example (e.g. on compiler explorer) that would be awesome!

[carlosgalvezp]

Also, regarding clang-analyzer-core checks, I have 2 tickets open:

https://github.com/llvm/llvm-project/issues/59588
https://github.com/llvm/llvm-project/issues/59589

[carlosgalvezp]

Reading through Github I found the associated ticket (it would be good if you could mention it in the commit message):
https://github.com/llvm/llvm-project/issues/61520

So if I understand correctly, what you are trying to do is continue to let the clang-analyzer-core checks run, and simply ignore the warnings coming from them when enabling warnings_as_errors. I'm not entirely convinced that's the best approach, it's hiding the real problem - that as a user one cannot disable clang-analyzer-core checks (which is being fixed in another patch by PiotrZSL). I also find strange to move error-handling logic away from the DiagnosticConsumer class out to the main ClangTidy class.

Would be good if @njames93 or @aaron.ballman could comment on this, given their experience.

[kiwixz]

Your understanding is correct, I'm currently just trying to hide them at the end (warnings-as-errors or not).
This is how it worked until 5d12b13b0b26bc58b02ee23c369da8b83240cceb (present in LLVM 16) which inadvertently bypass the filters with warnings-as-errors.

Note that I'm just trying to restore the previous behavior, while I'm sure PiotrZSL and others will make the whole thing more elegant for next releases.
My hope here is to fix LLVM 16 so we don't have errors that we explicitly disabled showing up, and only when enabling warnings-as-errors.

My patch does two things:

• deleting the two lines in ClangTidyDiagnosticConsumer.cpp which introduced the new and bad behavior
• adding two lines in ClangTidy.cpp to classify warnings as errors when appropriate in the YAML (this was the goal of the aforementioned commit)

[aaron.ballman]

In D146520#4221314, @carlosgalvezp wrote:

Reading through Github I found the associated ticket (it would be good if you could mention it in the commit message):
https://github.com/llvm/llvm-project/issues/61520

So if I understand correctly, what you are trying to do is continue to let the clang-analyzer-core checks run, and simply ignore the warnings coming from them when enabling warnings_as_errors. I'm not entirely convinced that's the best approach, it's hiding the real problem - that as a user one cannot disable clang-analyzer-core checks (which is being fixed in another patch by PiotrZSL). I also find strange to move error-handling logic away from the DiagnosticConsumer class out to the main ClangTidy class.

Would be good if @njames93 or @aaron.ballman could comment on this, given their experience.

Ideally, users should be able to disable the core checks from running at all (the reason users want to disable checks is both because of the diagnostics they produce and because of the cost of running the check itself). However, this is solving the issue accidentally introduced which is incremental forward progress.

I'd say this is tentatively okay, but definitely needs test coverage for the change. I'd hope that the fix to no longer run the core checks at all will obviate the need for this code, though, so it might be appropriate to add a FIXME comment about that.

[PiotrZSL]

In D146520#4224281, @aaron.ballman wrote:

Ideally, users should be able to disable the core checks from running at all (the reason users want to disable checks is both because of the diagnostics they produce and because of the cost of running the check itself).

I'm trying to enable this here: https://reviews.llvm.org/D146396, but problem is that I'm not sure what core checkers should be enabled by default. Simply Static Analysis config file does not show that.

[aaron.ballman]

In D146520#4224483, @PiotrZSL wrote:

In D146520#4224281, @aaron.ballman wrote:

Ideally, users should be able to disable the core checks from running at all (the reason users want to disable checks is both because of the diagnostics they produce and because of the cost of running the check itself).

I'm trying to enable this here: https://reviews.llvm.org/D146396, but problem is that I'm not sure what core checkers should be enabled by default. Simply Static Analysis config file does not show that.

CC @NoQ @xazax.hun for a question about what core checkers should be enabled by default in clang-tidy (and general awareness that this is about changes to how clang-tidy surfaces the static analyzer).

[tiagoma]

Applying this patch seems to fix issues with C++20 as well. See https://github.com/llvm/llvm-project/issues/61969
Can we get it in?

[carlosgalvezp]

Having another look and reading your comments I got a better understanding of the situation:

• YAML output did not have correct warnings as errors.
• Thus a fix was introduced in ClangTidyDiagnosticConsumer, however this was a too broad place to put the fix, breaking diagnostics output by command-line.
• Instead, the fix should have been put in a narrower scope, in the part of the code that deals with YAML output.
• This patch does exactly that.

So this patch is not really altering the original diagnostic behavior. In that sense I'm ok with it! I suppose it would have been clearer from a review perspective if first a revert was made and then re-landed with the correct code, but it's probably too much annoyance for such a small change.

[carlosgalvezp]

It would still be good to add some test that ensures this does not happen again in the future, though.

[kiwixz]

I'm not sure how to test this, is there any check dependency we can rely on with hardcoded tests ?

[kiwixz]

Ping.

[PiotrZSL]

Look into clang-tools-extra/test/clang-tidy/infrastructure/ for tests, probably you could base on one of them.

[PiotrZSL]

Add test & release notes.
Cleanup commit description.

clang-tools-extra/clang-tidy/ClangTidy.cpp

 
 void exportReplacements(const llvm::StringRef MainFilePath,
                         const std::vector<ClangTidyError> &Errors,
                         raw_ostream &OS) {
   TranslationUnitDiagnostics TUD;
   TUD.MainSourceFile = std::string(MainFilePath);
   for (const auto &Error : Errors) {
     tooling::Diagnostic Diag = Error;
+    if (Error.IsWarningAsError)
+      Diag.DiagLevel = tooling::Diagnostic::Error;
     TUD.Diagnostics.insert(TUD.Diagnostics.end(), Diag);
   }
 
   yaml::Output YAML(OS);
   YAML << TUD;
 }
 
 NamesAndOptions

clang-tools-extra/clang-tidy/ClangTidyDiagnosticConsumer.cpp

if (DiagLevel == DiagnosticsEngine::Error ||
       LastErrorRelatesToUserCode = true;
       LastErrorPassesLineFilter = true;
     } else if (DiagLevel == DiagnosticsEngine::Remark) {
       Level = ClangTidyError::Remark;
     }
 
     bool IsWarningAsError = DiagLevel == DiagnosticsEngine::Warning &&
                             Context.treatAsError(CheckName);
-    if (IsWarningAsError)
-      Level = ClangTidyError::Error;
     Errors.emplace_back(CheckName, Level, Context.getCurrentBuildDirectory(),
                         IsWarningAsError);
   }
 
   if (ExternalDiagEngine) {
     // If there is an external diagnostics engine, like in the
     // ClangTidyPluginAction case, forward the diagnostics to it.
     forwardDiagnostic(Info);

clang-tools-extra/docs/ReleaseNotes.rst

 - Support specifying `Checks` as a YAML list in the `.clang-tidy` configuration
   file.
 
 - Fix a potential crash when using the `--dump-config` option.
 
 - Support specifying `SystemHeaders` in the `.clang-tidy` configuration file,
   with the same functionality as the command-line option `--system-headers`.
 
+- `WarningsAsErrors` (`--warnings-as-errors=`) no longer promotes unlisted
+  warnings to errors. Only the warnings listed in `Checks` (`--checks=`) will
+  be promoted to errors. For custom error promotion, use `-Werror=<warning>`
+  on the compiler command-line, irrespective of `Checks` (`--checks=`) settings.
+
 New checks
 ^^^^^^^^^^
 
 - New :doc:`bugprone-multiple-new-in-one-expression
   <clang-tidy/checks/bugprone/multiple-new-in-one-expression>` check.
 
   Finds multiple ``new`` operator calls in a single expression, where the allocated
   memory by the first ``new`` may leak if the second allocation fails and throws exception.

clang-tools-extra/test/clang-tidy/infrastructure/warnings-as-errors-diagnostics.cpp

-// RUN: clang-tidy %s -checks='-*,llvm-namespace-comment,clang-diagnostic*' \
+// RUN: clang-tidy %s --checks='-*,llvm-namespace-comment,clang-diagnostic*' \
 // RUN:   -- -Wunused-variable 2>&1 \
 // RUN:   | FileCheck %s --check-prefix=CHECK-WARN -implicit-check-not='{{warning|error}}:'
-// RUN: not clang-tidy %s -checks='-*,llvm-namespace-comment,clang-diagnostic*' \
-// RUN:   -warnings-as-errors='clang-diagnostic*' -- -Wunused-variable 2>&1 \
+// RUN: not clang-tidy %s --checks='-*,llvm-namespace-comment,clang-diagnostic*' \
+// RUN:   --warnings-as-errors='clang-diagnostic*' -- -Wunused-variable 2>&1 \
 // RUN:   | FileCheck %s --check-prefix=CHECK-WERR -implicit-check-not='{{warning|error}}:'
-// RUN: not clang-tidy %s -checks='-*,llvm-namespace-comment,clang-diagnostic*' \
-// RUN:   -warnings-as-errors='clang-diagnostic*' -quiet -- -Wunused-variable 2>&1 \
+// RUN: not clang-tidy %s --checks='-*,llvm-namespace-comment,clang-diagnostic*' \
+// RUN:   --warnings-as-errors='clang-diagnostic*' -quiet -- -Wunused-variable 2>&1 \
 // RUN:   | FileCheck %s --check-prefix=CHECK-WERR-QUIET -implicit-check-not='{{warning|error}}:'
+// RUN: clang-tidy %s --checks='-*,llvm-namespace-comment' --warnings-as-errors=* -- \
+// RUN:   -Wunused-variable 2>&1 | FileCheck %s --check-prefix=CHECK-SUPPRESSED \
+// RUN:   -implicit-check-not='{{warning|error}}:'
 
 void f() { int i; }
 // CHECK-WARN: warning: unused variable 'i' [clang-diagnostic-unused-variable]
 // CHECK-WERR: error: unused variable 'i' [clang-diagnostic-unused-variable,-warnings-as-errors]
 // CHECK-WERR-QUIET: error: unused variable 'i' [clang-diagnostic-unused-variable,-warnings-as-errors]
 
 // CHECK-WARN-NOT: treated as
 // CHECK-WERR: 1 warning treated as error
 // CHECK-WERR-QUIET-NOT: treated as
+
+// CHECK-SUPPRESSED-NOT: unused variable 'i'
+// CHECK-SUPPRESSED: 1 warning generated.
+// CHECK-SUPPRESSED: Suppressed 1 warnings (1 with check filters).