There are a lot of common pattern that results in testcases that look like they
are checking something, but in fact they silently hide failures, leading to
accidental false negatives.
CHECK: legitimate test CHECK gotcha 1 CHECK : gotcha 2 CHECK-SAME-DAG: gotcha 3 CHECKNEXT: gotcha 4 CHECKDAG: gotcha 5
This patch attempts to catch 1 & 2, but it unfortunately still has a lot of
false positives, as it triggers on comments in testcases that refer to the
CHECKs themselves. The vast majority of false positives come from this
diagnostic triggering on RUN lines, which could be ameliorated (mostly) with a
precursor patch that quotes all the CHECK prefixes:
find -E llvm/test -regex ".*\.(c|cpp|m|ll|mir|yaml|s|td|test|txt)" -exec perl -p -i -e "s/check-prefix(es)?[= ]([^ \r\n]+)/check-prefix\1=\"\2\"/g" {} \;
But even with that, there are still ~455 "failures" this causes in llvm/test alone.
Is this worth moving forward on? If so, how else can we improve FileCheck to
prevent these kinds of issues?
I believe you need to change the order here. For example, CHECK-NEXT-NOT will match CHECK-NEXT and then be ignored. Likewise for CHECK-NOT-NEXT and CHECK-NOT.
If FileCheck's test suite doesn't already cover cases like that, now would be a good time to extend it. It might not be a bad idea to move all that into a parent patch as the refactoring and tests seems worthwhile independently of the new diagnostic.