[LNT] Python 3 support: run tests with UTF-8 encoding

[thopre]

Summary

Click requires to be invoked with a default encoding of UTF-8. If not,
it exists the program with an error. This commit sets the LANG
environment variable for running lit tests to the system locale if it is
a UTF-8 capable one or en_US.UTF-8 as a fallback. This allows the test
to run on systems where en_US.UTF-8 is not installed (such as some
container images) but the default locale is UTF-8 capable.

[hubert.reinterpretcast]

I don't think that this is a clear "win". Users may have UTF-8 capable locales not ending in ".UTF-8" and also not have a usable en_US.UTF-8 locale. It might be nice for some people, but this can actually break others. I don't see an issue with asking users to specify an appropriate value of LANG for their environment.

[thopre]

In D69048#1711272, @hubert.reinterpretcast wrote:

I don't think that this is a clear "win". Users may have UTF-8 capable locales not ending in ".UTF-8" and also not have a usable en_US.UTF-8 locale. It might be nice for some people, but this can actually break others. I don't see an issue with asking users to specify an appropriate value of LANG for their environment.

I can't quite remember but I think the problem is that lit starts from a clean environment to have consistent output (if using the user's environment error message might vary accross systems which would fail lots of tests). AFAIK LNT doesn't have localized error message but it fails to run if the locale is not UTF-8 (or perhaps any unicode locale).

[Sorry for the late answer, I missed that feedback.]

[hubert.reinterpretcast]

In D69048#1768965, @thopre wrote:

I can't quite remember but I think the problem is that lit starts from a clean environment to have consistent output (if using the user's environment error message might vary accross systems which would fail lots of tests). AFAIK LNT doesn't have localized error message but it fails to run if the locale is not UTF-8 (or perhaps any unicode locale).

That's my understanding regarding LNT as well; however, the "clean environment" seems to be what is being adjusted here based on the user's environment. My suggestion is to trust the user's environment by passing it through verbatim (since the patch already trusts the user if the locale name ends with .UTF-8). The user has to specify an appropriate environment anyway when running LNT outside of tests.

[thopre]

Unconditionally set LANG in lit environment to host's LANG

[hubert.reinterpretcast]

LGTM. Thanks.

[thopre]

Committed as e27299217d88ab3e0fc99fcaf83679142c2bb268

tests/lit.cfg

 config.target_triple = None
 
 src_root = os.path.join(config.test_source_root, '..')
 try:
     build_root = glob.glob('%s/build/lib.*' % src_root)[0]
 except:
     build_root = ''
 config.environment['PYTHONPATH'] = '%s:%s' % (build_root, src_root)
+# LNT needs a UTF-8 locale when running in Python 3 mode. Rely on the host
+# environment to provide it rather than assume a specific UTF-8 locale is
+# present.
+config.environment['LANG'] = os.environ['LANG']
 # Don't generate .pyc files when running tests.
 config.environment['PYTHONDONTWRITEBYTECODE'] = "1"
 config.environment['SUDO_CMD'] = ""
 config.environment['I'] = ""
 
 config.substitutions.append(('%src_root', src_root))
 config.substitutions.append(('%{src_root}', src_root))
 config.substitutions.append(('%{shared_inputs}', os.path.join(