Page MenuHomePhabricator

[lit] Don't fail when printing test output with special chars

Authored by jdenny on Oct 18 2019, 4:27 PM.



This addresses a UnicodeEncodeError when using Python 3.6.5 in Windows

Diff Detail

Event Timeline

jdenny created this revision.Oct 18 2019, 4:27 PM
Herald added a project: Restricted Project. · View Herald TranscriptOct 18 2019, 4:27 PM
Herald added a subscriber: delcypher. · View Herald Transcript
jdenny updated this revision to Diff 225773.Oct 19 2019, 3:45 PM

While the test suite was already passing for me in Windows and Ubuntu using either Python 2 or 3, it raised an exception when passing -a to lit (or if the test suite failed and managed to print output containing special characters) in Ubuntu when using Python 2. This change fixes that.

Also rebased onto a more recent master.

rnk accepted this revision.Oct 21 2019, 3:44 PM


This revision is now accepted and ready to land.Oct 21 2019, 3:44 PM
This revision was automatically updated to reflect the committed changes.

I have a failure on my bot for tools/llvm-ar/mri-utf8.test ; and this patch is making it worse.

Without this patch I have:

: 'RUN: at line 23';   env LANG=en_US.UTF-8 "/usr/bin/python" -c "assert open(u'\U000000A3.txt', 'rb').read() == b'contents\n'"
Exit Code: 1
Command Output (stderr):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in position 0: ordinal not in range(128)

I suspect en_US.UTF-8 is just not available on the system.

After this patch however the error becomes:

[3110/3111] Running the LLVM regression tests
-- Testing: 34143 tests, 64 workers --
FAIL: LLVM :: tools/llvm-ar/mri-utf8.test (32130 of 34143)
******************** TEST 'LLVM :: tools/llvm-ar/mri-utf8.test' FAILED ********************
Exception in thread Thread-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/", line 801, in __bootstrap_inner
  File "/usr/lib/python2.7/", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/", line 408, in _handle_results
    cache[job]._set(i, obj)
  File "/usr/lib/python2.7/multiprocessing/", line 577, in _set
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/", line 136, in <lambda>
    callback=lambda r, t=test: self._process_result(t, r))
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/", line 82, in _process_result
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/", line 194, in progress_callback
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/", line 52, in update
  File "/var/lib/buildkite-agent/builds/buildkite-6dd7db7bf5-rcq8r-1/mlir/llvm-coverage/llvm/utils/lit/lit/", line 85, in print_result
    out = out.decode(encoding=sys.stdout.encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 492: ordinal not in range(128)

And lit will just hang there forever.

Can you please try adding an errors='ignore' argument to the failing decode call? Or I can push a patch if that's easier. Thanks.