[lit] Don't fail when printing test output with special chars

[jdenny]

Summary

This addresses a UnicodeEncodeError when using Python 3.6.5 in Windows
10.

[jdenny]

While the test suite was already passing for me in Windows and Ubuntu using either Python 2 or 3, it raised an exception when passing -a to lit (or if the test suite failed and managed to print output containing special characters) in Ubuntu when using Python 2. This change fixes that.

Also rebased onto a more recent master.

[rnk]

lgtm

[mehdi_amini]

I have a failure on my bot for tools/llvm-ar/mri-utf8.test ; and this patch is making it worse.

Without this patch I have:

: 'RUN: at line 23';   env LANG=en_US.UTF-8 "/usr/bin/python" -c "assert open(u'\U000000A3.txt', 'rb').read() == b'contents\n'"
--
Exit Code: 1
 
Command Output (stderr):
--
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in position 0: ordinal not in range(128)
 
--

I suspect en_US.UTF-8 is just not available on the system.

After this patch however the error becomes:

[3110/3111] Running the LLVM regression tests
-- Testing: 34143 tests, 64 workers --
FAIL: LLVM :: tools/llvm-ar/mri-utf8.test (32130 of 34143)
******************** TEST 'LLVM :: tools/llvm-ar/mri-utf8.test' FAILED ********************
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 408, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 577, in _set
    self._callback(self._value)
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/run.py", line 136, in <lambda>
    callback=lambda r, t=test: self._process_result(t, r))
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/run.py", line 82, in _process_result
    self.progress_callback(test)
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/main.py", line 194, in progress_callback
    display.update(test)
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/display.py", line 52, in update
    self.print_result(test)
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/display.py", line 85, in print_result
    out = out.decode(encoding=sys.stdout.encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 492: ordinal not in range(128)

And lit will just hang there forever.

[jdenny]

Can you please try adding an errors='ignore' argument to the failing decode call? Or I can push a patch if that's easier. Thanks.

llvm/utils/lit/lit/display.py

def print_result(self, test):
                                      self.completed, self.numTests))
 
         # Show the test failure output, if requested.
         if (test.result.code.isFailure and self.opts.showOutput) or \
            self.opts.showAllOutput:
             if test.result.code.isFailure:
                 print("%s TEST '%s' FAILED %s" % ('*'*20, test.getFullName(),
                                                   '*'*20))
-            print(test.result.output)
+            out = test.result.output
+            # Encode/decode so that, when using Python 3.6.5 in Windows 10,
+            # print(out) doesn't raise UnicodeEncodeError if out contains
+            # special characters.  However, Python 2 might try to decode
+            # as part of the encode call if out is already encoded, so skip
+            # encoding if it raises UnicodeDecodeError.
+            if sys.stdout.encoding:
+                try:
+                    out = out.encode(encoding=sys.stdout.encoding,
+                                     errors="replace")
+                except UnicodeDecodeError:
+                    pass
+                out = out.decode(encoding=sys.stdout.encoding)
+            print(out)
             print("*" * 20)
 
         # Report test metrics, if present.
         if test.result.metrics:
             print("%s TEST '%s' RESULTS %s" % ('*'*10, test.getFullName(),
                                                '*'*10))
             items = sorted(test.result.metrics.items())
             for metric_name, value in items:

llvm/utils/lit/tests/Inputs/shtest-shell/stdout-encoding.txt

+# Check that lit doesn't fail when printing special characters in its test
+# results.
+
+# RUN: cat diff-in.bin
+
+# Fail so lit will print output.
+# RUN: false

llvm/utils/lit/tests/max-failures.py

 # Check the behavior of --max-failures option.
 #
 # RUN: not %{lit} -j 1 -v %{inputs}/max-failures > %t.out
 # RUN: not %{lit} --max-failures=1 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=2 -j 1 -v %{inputs}/max-failures >> %t.out
 # RUN: not %{lit} --max-failures=0 -j 1 -v %{inputs}/max-failures 2>> %t.out
 # RUN: FileCheck < %t.out %s
 #
 # END.
 
-# CHECK: Failing Tests (32)
+# CHECK: Failing Tests (33)
 # CHECK: Failing Tests (1)
 # CHECK: Failing Tests (2)
 # CHECK: error: argument --max-failures: requires positive integer, but found '0'

llvm/utils/lit/tests/shtest-shell.py

 # CHECK: FAIL: shtest-shell :: rm-error-3.txt
 # CHECK: *** TEST 'shtest-shell :: rm-error-3.txt' FAILED ***
 # CHECK: Exit Code: 1
 # CHECK: ***
 
 # CHECK: PASS: shtest-shell :: rm-unicode-0.txt
 # CHECK: PASS: shtest-shell :: sequencing-0.txt
 # CHECK: XFAIL: shtest-shell :: sequencing-1.txt
+
+# CHECK: FAIL: shtest-shell :: stdout-encoding.txt
+# CHECK: *** TEST 'shtest-shell :: stdout-encoding.txt' FAILED ***
+# CHECK: $ "cat" "diff-in.bin"
+# CHECK: # command output:
+# CHECK-NEXT: {{^.f.o.o.$}}
+# CHECK-NEXT: {{^.b.a.r..}}
+# CHECK-NEXT: {{^.b.a.z.$}}
+# CHECK-NOT: error
+# CHECK: $ "false"
+# CHECK: ***
+
 # CHECK: PASS: shtest-shell :: valid-shell.txt
-# CHECK: Failing Tests (32)
+# CHECK: Failing Tests (33)