[llvm test suite] Treat NOEXE test result as a failure

[DavidSpickett]

Summary

Currently the SVE bots have been crashing when trying to compile
some of the bitcode tests. This was only noticed recently by chance
but it is not reported as a failure on the bots.

This makes it so that NOEXE counts as a failed test. This seems to
make sense because NOEXE is similar to unresolved tests.

I also spot checked other test-suite bots and found that none
of them had existing NOEXE results. (I thought there might be some
that are failing to build but this is expected to happen)

In any case, tests are guarded with cmake files to enable them
only if you have the right tools. So I think a failure to compile
should be a failure.

[DavidSpickett]

See https://lab.llvm.org/buildbot/#/builders/197/builds/1432 where we get:

26:30 Unexpected test result output NOEXE 2203 expected passes13 test-suite

When you look in the logs we have a lot of missing test executables because of a compiler crash with simd_ops.

Testing Time: 484.23s
  Passed            : 2203
  Executable Missing: 1390

clang++: ../llvm/llvm/lib/Target/AArch64/AArch64ISelLowering.cpp:20021: llvm::SDValue llvm::AArch64TargetLowering::LowerToPredicatedOp(llvm::SDValue, llvm::SelectionDAG &, unsigned int) const: Assertion `VT.getFixedSizeInBits() <= Subtarget->getMinSVEVectorSizeInBits() && "Cannot use SVE to lower fixed length predicated op!"' failed.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
/usr/bin/make -f Fortran/UnitTests/fcvs21_f95/CMakeFiles/FM317.dir/build.make Fortran/UnitTests/fcvs21_f95/CMakeFiles/FM317.dir/build
0.	Program arguments: /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/stage1.install/bin/clang++ -DNDEBUG -mcpu=a64fx -mllvm -scalable-vectorization=preferred -mllvm -treat-scalable-fixed-error-as-warning=false -O3 -w -Werror=date-time -o CMakeFiles/simd_ops.dir/AArch64_halide_runtime.bc.o -c /home/tcwg-buildbot/worker/clang-aarch64-sve-vla/test/test-suite/Bitcode/simd_ops/AArch64_halide_runtime.bc
1.	Code generation

Which is its own issue ofc but this should surface it as a failure.

[sdesmalen]

Showing a failure on build failures is one of the things that we rely on the buildbots for, so I agree it makes sense to check for NOEXE. Thanks for the fix!

[DavidSpickett]

Will hold off on landing this until the test(s) in question are fixed. That way the bot remains useful for the normal build and test stages.

The other failure like this was a Fortran test that should be addressed by https://reviews.llvm.org/D121287.

[sdesmalen]

D121297 has been committed, so I think this patch should be good to land.

[DavidSpickett]

SVE and Fortran failures have been dealt with so this has been committed.

zorg/buildbot/commands/LitTestCommand.py

class LitLogObserver(LogLineObserver):
 
   # Regular expressions for verbose log start and stop markers. Verbose log
   # output always immediately follow a test.
   kTestVerboseLogStartRE = re.compile(r"""\*{4,80} TEST '(.*)' .*""")
   kTestVerboseLogStopRE = re.compile(r"""\*{10,80}""")
 
   # These are the codes for which we will include the log output in the buildbot
   # step results.
-  failingCodes = set(['FAIL', 'XPASS', 'KPASS', 'UNRESOLVED', 'TIMEOUT', 'ERROR'])
+  failingCodes = set(['FAIL', 'XPASS', 'KPASS', 'UNRESOLVED', 'TIMEOUT', 'ERROR', 'NOEXE'])
   # Regular expressions for start of summary marker.
   kStartSummaryRE = re.compile(r'^Failing Tests \(\d*\)$')
 
   def __init__(self, maxLogs=None, parseSummaryOnly=False):
     super().__init__()
     self.resultCounts = {}
     self.maxLogs = maxLogs
     self.numLogs = 0
resultNames = {'FAIL':'unexpected failures',
                  'KFAIL':'known failures',
                  'KPASS':'unknown passes',
                  'UNRESOLVED':'unresolved testcases',
                  'UNTESTED':'untested testcases',
                  'REGRESSED':'runtime performance regression',
                  'IMPROVED':'runtime performance improvement',
                  'UNSUPPORTED':'unsupported tests',
                  'SKIPPED':'skipped tests',
-                 'TIMEOUT':'timeout waiting for results'}
+                 'TIMEOUT':'timeout waiting for results',
+                 'NOEXE':'test executable is missing'}
 
   def __init__(self, ignore=[], flaky=[], max_logs=20, parseSummaryOnly=False,
                *args, **kwargs):
     super().__init__(*args, **kwargs)
     self.maxLogs = int(max_logs)
     self.logObserver = LitLogObserver(self.maxLogs, parseSummaryOnly)
     self.addFactoryArguments(max_logs=max_logs)
     self.addFactoryArguments(parseSummaryOnly=parseSummaryOnly)