[libc++] Create a small DSL for defining Lit features and parameters

[ldionne]

Summary

This allows defining Lit features that can be enabled or disabled based
on compiler support, and parameters that are passed on the command line.

The main benefits are:

• Feature detection is entirely based on the substitutions provided in the TestingConfig object, which is simpler and decouples it from the complicated compiler emulation infrastructure.
• The syntax is declarative, which makes it easy to see what features and parameters are accepted by the test suite. This is significantly less entangled than the current config.py logic.
• Since feature detection is based on substitutions, it works really well on top of the new format, and custom Lit configurations can be created easily without being based on config.py.

[ldionne]

This is in itself a NFC change -- only https://reviews.llvm.org/D78382 has a real impact on the config. If there are no comments, I'll go ahead and ship this tomorrow so I can see if the added test passes on Windows. It is expected to, but if it doesn't, that will be valuable information to improve this patch.

[EricWF]

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

34

A short docstring for each class would be a nice addition.

72

I think the python style guidlines wants 2 spaces between class definitions.

92

Can we watch a more specific exception as to not swallow other errors that may occur?

libcxx/utils/libcxx/test/dsl.py

51

I don't know that this is correct with GCC.
And it doesn't handle -Wno-foo flags.

IIRC the compiler may only report unknown flags as error if another error occurs during compilation.

51

Also, just use os.devnul as the input file, because pipes don't work with popen.

70

Last I checked locales didn't always have a single canonical spelling. Are you handling this case?

[ldionne]

Address review comments. Also, harden checking of locales so it should work
on Windows too.

libcxx/utils/libcxx/test/dsl.py

51

I don't know that this is correct with GCC. And it doesn't handle -Wno-foo flags.

It is correct with GCC outside of -Wno-foo flags, but you are correct that it doesn't handle -Wno-foo flags correctly. As-is, it will report that all -Wno-foo flags are supported on GCC. I went ahead and filed https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94745 because I thought it was a GCC bug, but Jonathan Wakely nicely explained to me how this was by design. He also suggested using g++ -Q --help=warnings (we should have that for Clang!).

I was going to go ahead and implement the g++ -Q --help=warnings solution, but then it occurred to me that if GCC reports they "support" a -Wno-foo flag when they don't (and they simply ignore it), then why is it a problem for the test suite to add that warning flag? In other words, why would we want to *not* add a flag that gets ignored? I believe that behavior is more in line with their intent when they made that design choice.

For this reason, I'll suggest we leave it as-is, but if you feel strongly about it, please explain and I can use g++ -Q --help=warnings on GCC to check for supported -Wno-foo flags.

70

This will be handled through trying various different locale names when setting up each locale's feature. This will be in an upcoming patch.

[ldionne]

I'm going to go ahead and ship this now so I can see whether it breaks anything.

[davezarzycki]

In D78381#2005219, @ldionne wrote:

I'm going to go ahead and ship this now so I can see whether it breaks anything.

This breaks Fedora 32 (x86_64) building with the system clang/lld (version 10.0):

[0+1] Running all regression tests
llvm-lit: /home/dave/s/lp/libcxxabi/test/lit.cfg:64: note: Using configuration variant: libcxxabi
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:307: note: inferred use_system_cxx_lib as: None
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:333: note: inferred use_clang_verify as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:342: note: enabling thread-safety annotations
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:487: note: inferred language dialect as: c++2a
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_FR.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale ru_RU.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale zh_CN.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_CA.ISO8859-1 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale cs_CZ.ISO8859-2 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:389: note: inferred long_tests as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:153: note: Using compiler: /usr/bin/clang++
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:154: note: Using flags: ['-v']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:158: note: Using compile flags: ['-Werror=thread-safety', '-DLIBCXXABI_NO_TIMER', '-D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS', '-funwind-tables', '-std=c++2a', '-nostdinc++', '-I/home/dave/s/lp/libcxx/include', '-I/home/dave/s/lp/libcxxabi/include', '-DSTDC_FORMAT_MACROS', '-DSTDC_LIMIT_MACROS', '-DSTDC_CONSTANT_MACROS', '-I/home/dave/s/lp/libcxx/test/support', '-ftemplate-depth=270']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:161: note: Using warnings: ['-Wall', '-Wextra', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-unused-local-typedef']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:162: note: Using link flags: ['-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-nodefaultlibs', '/tmp/_update_lc/r/./lib64/libc++.a', '/tmp/_update_lc/r/./lib64/libc++abi.a', '-lm', '-lgcc_s', '-lgcc', '-lpthread', '-lrt', '-lc', '-lgcc_s', '-lgcc']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:164: note: Using available_features: ['long_tests', 'fdelayed-template-parsing', 'libc++', 'c++2a', 'diagnose-if-support', 'has-fblocks', '-faligned-allocation', 'clang-10', 'clang', 'locale.en_US.UTF-8', 'linux', 'clang-10.0.0', '-fsized-deallocation', 'thread-safety', 'libcxxabi-has-system-unwinder', 'fcoroutines-ts', 'modules-support', 'clang-10.0']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:170: note: Adding environment variables: {}
llvm-lit: /home/dave/s/lp/libcxxabi/test/lit.cfg:77: note: Using the new libc++ testing format
llvm-lit: /home/dave/s/lp/libcxx/test/lit.cfg:49: note: Using configuration variant: libcxx
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:307: note: inferred use_system_cxx_lib as: None
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:333: note: inferred use_clang_verify as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:342: note: enabling thread-safety annotations
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:487: note: inferred language dialect as: c++2a
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_FR.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale ru_RU.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale zh_CN.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_CA.ISO8859-1 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale cs_CZ.ISO8859-2 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:389: note: inferred long_tests as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:153: note: Using compiler: /usr/bin/clang++
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:154: note: Using flags: ['-v']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:158: note: Using compile flags: ['-Werror=thread-safety', '-std=c++2a', '-include', '/home/dave/s/lp/libcxx/test/support/nasty_macros.h', '-nostdinc++', '-I/home/dave/s/lp/libcxx/include', '-I/tmp/_update_lc/r/projects/libcxx/include/c++build', '-DSTDC_FORMAT_MACROS', '-DSTDC_LIMIT_MACROS', '-DSTDC_CONSTANT_MACROS', '-I/home/dave/s/lp/libcxx/test/support', '-ftemplate-depth=270']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:161: note: Using warnings: ['-Wall', '-Wextra', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-unused-local-typedef']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:162: note: Using link flags: ['-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-nodefaultlibs', '-lc++experimental', '/tmp/_update_lc/r/./lib64/libc++.a', '-lc++abi', '-lm', '-lgcc_s', '-lgcc', '-lpthread', '-lrt', '-lc', '-lgcc_s', '-lgcc']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:164: note: Using available_features: ['modules-support', 'clang-10.0', 'fdelayed-template-parsing', 'diagnose-if-support', 'has-fblocks', 'clang-10', 'libatomic', '-faligned-allocation', 'clang', 'locale.en_US.UTF-8', 'c++experimental', 'long_tests', 'libc++', 'c++2a', 'linux', 'clang-10.0.0', 'thread-safety', 'fcoroutines-ts', '-fsized-deallocation']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:170: note: Adding environment variables: {}
llvm-lit: /home/dave/s/lp/libcxx/test/lit.cfg:62: note: Using the new libc++ testing format
llvm-lit: /tmp/_update_lc/r/bin/../../../../home/dave/s/lp/llvm/utils/lit/lit/TestingConfig.py:100: fatal: unable to parse config file '/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg', traceback: Traceback (most recent call last):

File "/tmp/_update_lc/r/bin/../../../../home/dave/s/lp/llvm/utils/lit/lit/TestingConfig.py", line 89, in load_from_path
  exec(compile(data, path, 'exec'), cfg_globals, None)
File "/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg", line 13, in <module>
  escaped = [(k.replace('%{', '%{escaped_'), base64.b64encode(v)) for (k, v) in config.substitutions]
File "/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg", line 13, in <listcomp>
  escaped = [(k.replace('%{', '%{escaped_'), base64.b64encode(v)) for (k, v) in config.substitutions]
File "/usr/lib64/python3.8/base64.py", line 58, in b64encode
  encoded = binascii.b2a_base64(s, newline=False)

TypeError: a bytes-like object is required, not 'str'

FAILED: CMakeFiles/check-all

[tpopp]

In D78381#2006266, @davezarzycki wrote:

In D78381#2005219, @ldionne wrote:

I'm going to go ahead and ship this now so I can see whether it breaks anything.

This breaks Fedora 32 (x86_64) building with the system clang/lld (version 10.0):

[0+1] Running all regression tests
llvm-lit: /home/dave/s/lp/libcxxabi/test/lit.cfg:64: note: Using configuration variant: libcxxabi
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:307: note: inferred use_system_cxx_lib as: None
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:333: note: inferred use_clang_verify as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:342: note: enabling thread-safety annotations
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:487: note: inferred language dialect as: c++2a
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_FR.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale ru_RU.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale zh_CN.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_CA.ISO8859-1 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale cs_CZ.ISO8859-2 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:389: note: inferred long_tests as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:153: note: Using compiler: /usr/bin/clang++
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:154: note: Using flags: ['-v']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:158: note: Using compile flags: ['-Werror=thread-safety', '-DLIBCXXABI_NO_TIMER', '-D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS', '-funwind-tables', '-std=c++2a', '-nostdinc++', '-I/home/dave/s/lp/libcxx/include', '-I/home/dave/s/lp/libcxxabi/include', '-DSTDC_FORMAT_MACROS', '-DSTDC_LIMIT_MACROS', '-DSTDC_CONSTANT_MACROS', '-I/home/dave/s/lp/libcxx/test/support', '-ftemplate-depth=270']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:161: note: Using warnings: ['-Wall', '-Wextra', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-unused-local-typedef']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:162: note: Using link flags: ['-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-nodefaultlibs', '/tmp/_update_lc/r/./lib64/libc++.a', '/tmp/_update_lc/r/./lib64/libc++abi.a', '-lm', '-lgcc_s', '-lgcc', '-lpthread', '-lrt', '-lc', '-lgcc_s', '-lgcc']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:164: note: Using available_features: ['long_tests', 'fdelayed-template-parsing', 'libc++', 'c++2a', 'diagnose-if-support', 'has-fblocks', '-faligned-allocation', 'clang-10', 'clang', 'locale.en_US.UTF-8', 'linux', 'clang-10.0.0', '-fsized-deallocation', 'thread-safety', 'libcxxabi-has-system-unwinder', 'fcoroutines-ts', 'modules-support', 'clang-10.0']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:170: note: Adding environment variables: {}
llvm-lit: /home/dave/s/lp/libcxxabi/test/lit.cfg:77: note: Using the new libc++ testing format
llvm-lit: /home/dave/s/lp/libcxx/test/lit.cfg:49: note: Using configuration variant: libcxx
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:307: note: inferred use_system_cxx_lib as: None
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:333: note: inferred use_clang_verify as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:342: note: enabling thread-safety annotations
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:487: note: inferred language dialect as: c++2a
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_FR.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale ru_RU.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale zh_CN.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_CA.ISO8859-1 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale cs_CZ.ISO8859-2 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:389: note: inferred long_tests as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:153: note: Using compiler: /usr/bin/clang++
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:154: note: Using flags: ['-v']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:158: note: Using compile flags: ['-Werror=thread-safety', '-std=c++2a', '-include', '/home/dave/s/lp/libcxx/test/support/nasty_macros.h', '-nostdinc++', '-I/home/dave/s/lp/libcxx/include', '-I/tmp/_update_lc/r/projects/libcxx/include/c++build', '-DSTDC_FORMAT_MACROS', '-DSTDC_LIMIT_MACROS', '-DSTDC_CONSTANT_MACROS', '-I/home/dave/s/lp/libcxx/test/support', '-ftemplate-depth=270']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:161: note: Using warnings: ['-Wall', '-Wextra', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-unused-local-typedef']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:162: note: Using link flags: ['-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-nodefaultlibs', '-lc++experimental', '/tmp/_update_lc/r/./lib64/libc++.a', '-lc++abi', '-lm', '-lgcc_s', '-lgcc', '-lpthread', '-lrt', '-lc', '-lgcc_s', '-lgcc']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:164: note: Using available_features: ['modules-support', 'clang-10.0', 'fdelayed-template-parsing', 'diagnose-if-support', 'has-fblocks', 'clang-10', 'libatomic', '-faligned-allocation', 'clang', 'locale.en_US.UTF-8', 'c++experimental', 'long_tests', 'libc++', 'c++2a', 'linux', 'clang-10.0.0', 'thread-safety', 'fcoroutines-ts', '-fsized-deallocation']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:170: note: Adding environment variables: {}
llvm-lit: /home/dave/s/lp/libcxx/test/lit.cfg:62: note: Using the new libc++ testing format
llvm-lit: /tmp/_update_lc/r/bin/../../../../home/dave/s/lp/llvm/utils/lit/lit/TestingConfig.py:100: fatal: unable to parse config file '/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg', traceback: Traceback (most recent call last):

File "/tmp/_update_lc/r/bin/../../../../home/dave/s/lp/llvm/utils/lit/lit/TestingConfig.py", line 89, in load_from_path
  exec(compile(data, path, 'exec'), cfg_globals, None)
File "/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg", line 13, in <module>
  escaped = [(k.replace('%{', '%{escaped_'), base64.b64encode(v)) for (k, v) in config.substitutions]
File "/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg", line 13, in <listcomp>
  escaped = [(k.replace('%{', '%{escaped_'), base64.b64encode(v)) for (k, v) in config.substitutions]
File "/usr/lib64/python3.8/base64.py", line 58, in b64encode
  encoded = binascii.b2a_base64(s, newline=False)

TypeError: a bytes-like object is required, not 'str'

FAILED: CMakeFiles/check-all

Just a note, this seems due to incompatibilities between python2 and python3. Python2 expects strings like given here while Python3 expects the bytes-like object.

[davezarzycki]

In D78381#2007451, @tpopp wrote:

In D78381#2006266, @davezarzycki wrote:

In D78381#2005219, @ldionne wrote:

I'm going to go ahead and ship this now so I can see whether it breaks anything.

This breaks Fedora 32 (x86_64) building with the system clang/lld (version 10.0):

[0+1] Running all regression tests
llvm-lit: /home/dave/s/lp/libcxxabi/test/lit.cfg:64: note: Using configuration variant: libcxxabi
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:307: note: inferred use_system_cxx_lib as: None
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:333: note: inferred use_clang_verify as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:342: note: enabling thread-safety annotations
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:487: note: inferred language dialect as: c++2a
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_FR.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale ru_RU.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale zh_CN.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_CA.ISO8859-1 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale cs_CZ.ISO8859-2 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:389: note: inferred long_tests as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:153: note: Using compiler: /usr/bin/clang++
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:154: note: Using flags: ['-v']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:158: note: Using compile flags: ['-Werror=thread-safety', '-DLIBCXXABI_NO_TIMER', '-D_LIBCPP_ENABLE_CXX17_REMOVED_UNEXPECTED_FUNCTIONS', '-funwind-tables', '-std=c++2a', '-nostdinc++', '-I/home/dave/s/lp/libcxx/include', '-I/home/dave/s/lp/libcxxabi/include', '-DSTDC_FORMAT_MACROS', '-DSTDC_LIMIT_MACROS', '-DSTDC_CONSTANT_MACROS', '-I/home/dave/s/lp/libcxx/test/support', '-ftemplate-depth=270']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:161: note: Using warnings: ['-Wall', '-Wextra', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-unused-local-typedef']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:162: note: Using link flags: ['-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-nodefaultlibs', '/tmp/_update_lc/r/./lib64/libc++.a', '/tmp/_update_lc/r/./lib64/libc++abi.a', '-lm', '-lgcc_s', '-lgcc', '-lpthread', '-lrt', '-lc', '-lgcc_s', '-lgcc']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:164: note: Using available_features: ['long_tests', 'fdelayed-template-parsing', 'libc++', 'c++2a', 'diagnose-if-support', 'has-fblocks', '-faligned-allocation', 'clang-10', 'clang', 'locale.en_US.UTF-8', 'linux', 'clang-10.0.0', '-fsized-deallocation', 'thread-safety', 'libcxxabi-has-system-unwinder', 'fcoroutines-ts', 'modules-support', 'clang-10.0']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:170: note: Adding environment variables: {}
llvm-lit: /home/dave/s/lp/libcxxabi/test/lit.cfg:77: note: Using the new libc++ testing format
llvm-lit: /home/dave/s/lp/libcxx/test/lit.cfg:49: note: Using configuration variant: libcxx
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:307: note: inferred use_system_cxx_lib as: None
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:333: note: inferred use_clang_verify as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:342: note: enabling thread-safety annotations
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:487: note: inferred language dialect as: c++2a
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_FR.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale ru_RU.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale zh_CN.UTF-8 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale fr_CA.ISO8859-1 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/target_info.py:84: warning: The locale cs_CZ.ISO8859-2 is not supported by your platform. Some tests will be unsupported.
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:389: note: inferred long_tests as: True
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:153: note: Using compiler: /usr/bin/clang++
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:154: note: Using flags: ['-v']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:158: note: Using compile flags: ['-Werror=thread-safety', '-std=c++2a', '-include', '/home/dave/s/lp/libcxx/test/support/nasty_macros.h', '-nostdinc++', '-I/home/dave/s/lp/libcxx/include', '-I/tmp/_update_lc/r/projects/libcxx/include/c++build', '-DSTDC_FORMAT_MACROS', '-DSTDC_LIMIT_MACROS', '-DSTDC_CONSTANT_MACROS', '-I/home/dave/s/lp/libcxx/test/support', '-ftemplate-depth=270']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:161: note: Using warnings: ['-Wall', '-Wextra', '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER', '-Wuser-defined-warnings', '-Wshadow', '-Wno-unused-command-line-argument', '-Wno-attributes', '-Wno-pessimizing-move', '-Wno-c++11-extensions', '-Wno-user-defined-literals', '-Wno-noexcept-type', '-Wsign-compare', '-Wunused-variable', '-Wunused-parameter', '-Wunreachable-code', '-Wno-unused-local-typedef']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:162: note: Using link flags: ['-L/tmp/_update_lc/r/./lib64', '-Wl,-rpath,/tmp/_update_lc/r/./lib64', '-nodefaultlibs', '-lc++experimental', '/tmp/_update_lc/r/./lib64/libc++.a', '-lc++abi', '-lm', '-lgcc_s', '-lgcc', '-lpthread', '-lrt', '-lc', '-lgcc_s', '-lgcc']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:164: note: Using available_features: ['modules-support', 'clang-10.0', 'fdelayed-template-parsing', 'diagnose-if-support', 'has-fblocks', 'clang-10', 'libatomic', '-faligned-allocation', 'clang', 'locale.en_US.UTF-8', 'c++experimental', 'long_tests', 'libc++', 'c++2a', 'linux', 'clang-10.0.0', 'thread-safety', 'fcoroutines-ts', '-fsized-deallocation']
llvm-lit: /home/dave/s/lp/libcxx/utils/libcxx/test/config.py:170: note: Adding environment variables: {}
llvm-lit: /home/dave/s/lp/libcxx/test/lit.cfg:62: note: Using the new libc++ testing format
llvm-lit: /tmp/_update_lc/r/bin/../../../../home/dave/s/lp/llvm/utils/lit/lit/TestingConfig.py:100: fatal: unable to parse config file '/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg', traceback: Traceback (most recent call last):

File "/tmp/_update_lc/r/bin/../../../../home/dave/s/lp/llvm/utils/lit/lit/TestingConfig.py", line 89, in load_from_path
  exec(compile(data, path, 'exec'), cfg_globals, None)
File "/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg", line 13, in <module>
  escaped = [(k.replace('%{', '%{escaped_'), base64.b64encode(v)) for (k, v) in config.substitutions]
File "/home/dave/s/lp/libcxx/test/libcxx/selftest/dsl/lit.local.cfg", line 13, in <listcomp>
  escaped = [(k.replace('%{', '%{escaped_'), base64.b64encode(v)) for (k, v) in config.substitutions]
File "/usr/lib64/python3.8/base64.py", line 58, in b64encode
  encoded = binascii.b2a_base64(s, newline=False)

TypeError: a bytes-like object is required, not 'str'

FAILED: CMakeFiles/check-all

Just a note, this seems due to incompatibilities between python2 and python3. Python2 expects strings like given here while Python3 expects the bytes-like object.

Right. Python 2.x was EOLed January 1st, 2020. See: https://www.python.org/dev/peps/pep-0373/#release-schedule

I'm going to revert this due to lack of Python 3 testing.

[ldionne]

In D78381#2007487, @davezarzycki wrote:

Right. Python 2.x was EOLed January 1st, 2020. See: https://www.python.org/dev/peps/pep-0373/#release-schedule

I'm going to revert this due to lack of Python 3 testing.

Thanks for the heads up. Note that the LLVM community decided to keep supporting Python 2 until 2021 AFAICT, which means that we need to support both right now. That's a poor decision IMO because it leads to these situations.

Regardless, I added support for Python3 and I'm going to recommit this. Thanks!

libcxx/test/libcxx/selftest/dsl/dsl.sh.py

+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+# RUN: %{python} %s '%{cxx}' '%{exec}' '%T' '%S'
+# END.
+
+import copy
+import os
+import platform
+import sys
+import unittest
+from os.path import dirname
+
+# Allow importing 'lit' and the 'libcxx' module. Make sure we put the lit
+# path first so we don't find any system-installed version.
+monorepoRoot = dirname(dirname(dirname(dirname(dirname(dirname(__file__))))))
+print monorepoRoot
+sys.path = [os.path.join(monorepoRoot, 'libcxx', 'utils'),
+            os.path.join(monorepoRoot, 'llvm', 'utils', 'lit')] + sys.path
+import libcxx.test.dsl as dsl
+import lit.LitConfig
+
+# Steal some parameters from the config running this test so that we can
+# bootstrap our own TestingConfig.
+SOURCE_ROOT = sys.argv.pop()
+EXEC_PATH = sys.argv.pop()
+EXEC = sys.argv.pop()
+CXX = sys.argv.pop()
+
+class SetupConfigs(unittest.TestCase):
+    def setUp(self):
+        """
+        Create a fake TestingConfig that can be populated however we wish for
+        the purpose of running unit tests below. We pre-populate it with the
+        minimum required substitutions.
+        """
+        self.litConfig = lit.LitConfig.LitConfig(
+            progname='lit',
+            path=[],
+            quiet=False,
+            useValgrind=False,
+            valgrindLeakCheck=False,
+            valgrindArgs=[],
+            noExecute=False,
+            debug=False,
+            isWindows=platform.system() == 'Windows',
+            params={})
+
+        self.config = lit.TestingConfig.TestingConfig.fromdefaults(self.litConfig)
+        self.config.test_source_root = SOURCE_ROOT
+        self.config.test_exec_root = EXEC_PATH
+        self.config.substitutions = [
+            ('%{cxx}', CXX),
+            ('%{compile_flags}', ''),
+            ('%{link_flags}', ''),
+            ('%{flags}', ''),
+            ('%{exec}', EXEC)
+        ]
+
+    def getSubstitution(self, substitution):
+        """
+        Return a given substitution from the TestingConfig. It is an error if
+        there is no such substitution.
+        """
+        found = [x for (s, x) in self.config.substitutions if s == substitution]
+        assert len(found) == 1
+        return found[0]
+
+class TestHasCompileFlag(SetupConfigs):
+    def test_no_flag_should_work(self):
+        self.assertTrue(dsl.hasCompileFlag(self.config, ''))
+
+    def test_flag_exists(self):
+        self.assertTrue(dsl.hasCompileFlag(self.config, '-O1'))
+
+    def test_nonexistent_flag(self):
+        self.assertFalse(dsl.hasCompileFlag(self.config, '-this_is_not_a_flag_any_compiler_has'))
+
+    def test_multiple_flags(self):
+        self.assertTrue(dsl.hasCompileFlag(self.config, '-O1 -Dhello'))
+
+class TestHasLocale(SetupConfigs):
+    # It's really hard to test that a system has a given locale, so at least
+    # make sure we don't explode when we try to check it.
+    def test_doesnt_explode(self):
+        try:
+            dsl.hasLocale(self.config, 'en_US.UTF-8')
+        except Exception:
+            self.fail("checking for hasLocale should not explode")
+
+    def test_nonexistent_locale(self):
+        self.assertFalse(dsl.hasLocale(self.config, 'for_sure_this_is_not_an_existing_locale'))
+
+class TestCompilerMacros(SetupConfigs):
+    def test_basic(self):
+        macros = dsl.compilerMacros(self.config)
+        self.assertIsInstance(macros, dict)
+        self.assertGreater(len(macros), 0)
+        for (k, v) in macros.items():
+            self.assertIsInstance(k, str)
+            self.assertIsInstance(v, str)
+
+    def test_no_flag(self):
+        macros = dsl.compilerMacros(self.config)
+        self.assertIn('__cplusplus', macros.keys())
+
+    def test_empty_flag(self):
+        macros = dsl.compilerMacros(self.config, '')
+        self.assertIn('__cplusplus', macros.keys())
+
+    def test_with_flag(self):
+        macros = dsl.compilerMacros(self.config, '-DFOO=3')
+        self.assertIn('__cplusplus', macros.keys())
+        self.assertEqual(macros['FOO'], '3')
+
+    def test_with_flags(self):
+        macros = dsl.compilerMacros(self.config, '-DFOO=3 -DBAR=hello')
+        self.assertIn('__cplusplus', macros.keys())
+        self.assertEqual(macros['FOO'], '3')
+        self.assertEqual(macros['BAR'], 'hello')
+
+class TestFeatureTestMacros(SetupConfigs):
+    def test_basic(self):
+        macros = dsl.featureTestMacros(self.config)
+        self.assertIsInstance(macros, dict)
+        self.assertGreater(len(macros), 0)
+        for (k, v) in macros.items():
+            self.assertIsInstance(k, str)
+            self.assertIsInstance(v, int)
+
+class TestFeature(SetupConfigs):
+    def test_trivial(self):
+        feature = dsl.Feature(name='name')
+        origSubstitutions = copy.deepcopy(self.config.substitutions)
+        self.assertTrue(feature.isSupported(self.config))
+        feature.enableIn(self.config)
+        self.assertEqual(origSubstitutions, self.config.substitutions)
+        self.assertIn('name', self.config.available_features)
+
+    def test_name_can_be_a_callable(self):
+        feature = dsl.Feature(name=lambda cfg: (self.assertIs(self.config, cfg), 'name')[1])
+        assert feature.isSupported(self.config)
+        feature.enableIn(self.config)
+        self.assertIn('name', self.config.available_features)
+
+    def test_adding_compile_flag(self):
+        feature = dsl.Feature(name='name', compileFlag='-foo')
+        origLinkFlags = copy.deepcopy(self.getSubstitution('%{link_flags}'))
+        assert feature.isSupported(self.config)
+        feature.enableIn(self.config)
+        self.assertIn('name', self.config.available_features)
+        self.assertIn('-foo', self.getSubstitution('%{compile_flags}'))
+        self.assertEqual(origLinkFlags, self.getSubstitution('%{link_flags}'))
+
+    def test_adding_link_flag(self):
+        feature = dsl.Feature(name='name', linkFlag='-foo')
+        origCompileFlags = copy.deepcopy(self.getSubstitution('%{compile_flags}'))
+        assert feature.isSupported(self.config)
+        feature.enableIn(self.config)
+        self.assertIn('name', self.config.available_features)
+        self.assertIn('-foo', self.getSubstitution('%{link_flags}'))
+        self.assertEqual(origCompileFlags, self.getSubstitution('%{compile_flags}'))
+
+    def test_adding_both_flags(self):
+        feature = dsl.Feature(name='name', compileFlag='-hello', linkFlag='-world')
+        assert feature.isSupported(self.config)
+        feature.enableIn(self.config)
+        self.assertIn('name', self.config.available_features)
+
+        self.assertIn('-hello', self.getSubstitution('%{compile_flags}'))
+        self.assertNotIn('-world', self.getSubstitution('%{compile_flags}'))
+
+        self.assertIn('-world', self.getSubstitution('%{link_flags}'))
+        self.assertNotIn('-hello', self.getSubstitution('%{link_flags}'))
+
+    def test_unsupported_feature(self):
+        feature = dsl.Feature(name='name', when=lambda _: False)
+        self.assertFalse(feature.isSupported(self.config))
+        # Also make sure we assert if we ever try to add it to a config
+        self.assertRaises(AssertionError, lambda: feature.enableIn(self.config))
+
+    def test_is_supported_gets_passed_the_config(self):
+        feature = dsl.Feature(name='name', when=lambda cfg: (self.assertIs(self.config, cfg), True)[1])
+        self.assertTrue(feature.isSupported(self.config))
+
+class TestParameter(SetupConfigs):
+    def test_empty_name_should_blow_up(self):
+        self.assertRaises(ValueError, lambda: dsl.Parameter(name='', choices=['c++03'], type=str, help='', feature=lambda _: None))
+
+    def test_empty_choices_should_blow_up(self):
+        self.assertRaises(ValueError, lambda: dsl.Parameter(name='std', choices=[], type=str, help='', feature=lambda _: None))
+
+    def test_name_is_set_correctly(self):
+        param = dsl.Parameter(name='std', choices=['c++03'], type=str, help='', feature=lambda _: None)
+        self.assertEqual(param.name, 'std')
+
+    def test_no_value_provided_on_command_line_and_no_default_value(self):
+        param = dsl.Parameter(name='std', choices=['c++03'], type=str, help='', feature=lambda _: None)
+        self.assertRaises(ValueError, lambda: param.getFeature(self.config, self.litConfig.params))
+
+    def test_no_value_provided_on_command_line_and_default_value(self):
+        param = dsl.Parameter(name='std', choices=['c++03'], type=str, help='', default='c++03',
+                              feature=lambda std: dsl.Feature(name=std))
+        param.getFeature(self.config, self.litConfig.params).enableIn(self.config)
+        self.assertIn('c++03', self.config.available_features)
+
+    def test_value_provided_on_command_line_and_no_default_value(self):
+        self.litConfig.params['std'] = 'c++03'
+        param = dsl.Parameter(name='std', choices=['c++03'], type=str, help='',
+                              feature=lambda std: dsl.Feature(name=std))
+        param.getFeature(self.config, self.litConfig.params).enableIn(self.config)
+        self.assertIn('c++03', self.config.available_features)
+
+    def test_value_provided_on_command_line_and_default_value(self):
+        self.litConfig.params['std'] = 'c++11'
+        param = dsl.Parameter(name='std', choices=['c++03', 'c++11'], type=str, default='c++03', help='',
+                              feature=lambda std: dsl.Feature(name=std))
+        param.getFeature(self.config, self.litConfig.params).enableIn(self.config)
+        self.assertIn('c++11', self.config.available_features)
+        self.assertNotIn('c++03', self.config.available_features)
+
+    def test_feature_is_None(self):
+        self.litConfig.params['std'] = 'c++03'
+        param = dsl.Parameter(name='std', choices=['c++03'], type=str, help='',
+                              feature=lambda _: None)
+        feature = param.getFeature(self.config, self.litConfig.params)
+        self.assertIsNone(feature)
+
+    def test_boolean_value_parsed_from_trueish_string_parameter(self):
+        self.litConfig.params['enable_exceptions'] = "True"
+        param = dsl.Parameter(name='enable_exceptions', choices=[True, False], type=bool, help='',
+                              feature=lambda exceptions: None if exceptions else ValueError())
+        self.assertIsNone(param.getFeature(self.config, self.litConfig.params))
+
+    def test_boolean_value_from_true_boolean_parameter(self):
+        self.litConfig.params['enable_exceptions'] = True
+        param = dsl.Parameter(name='enable_exceptions', choices=[True, False], type=bool, help='',
+                              feature=lambda exceptions: None if exceptions else ValueError())
+        self.assertIsNone(param.getFeature(self.config, self.litConfig.params))
+
+    def test_boolean_value_parsed_from_falseish_string_parameter(self):
+        self.litConfig.params['enable_exceptions'] = "False"
+        param = dsl.Parameter(name='enable_exceptions', choices=[True, False], type=bool, help='',
+                              feature=lambda exceptions: None if exceptions else dsl.Feature(name="-fno-exceptions"))
+        param.getFeature(self.config, self.litConfig.params).enableIn(self.config)
+        self.assertIn('-fno-exceptions', self.config.available_features)
+
+    def test_boolean_value_from_false_boolean_parameter(self):
+        self.litConfig.params['enable_exceptions'] = False
+        param = dsl.Parameter(name='enable_exceptions', choices=[True, False], type=bool, help='',
+                              feature=lambda exceptions: None if exceptions else dsl.Feature(name="-fno-exceptions"))
+        param.getFeature(self.config, self.litConfig.params).enableIn(self.config)
+        self.assertIn('-fno-exceptions', self.config.available_features)
+
+
+if __name__ == '__main__':
+    unittest.main()

libcxx/test/libcxx/selftest/dsl/lit.local.cfg

+# The tests in this directory need to run Python
+import sys
+config.substitutions.append(('%{python}', sys.executable))

libcxx/utils/libcxx/test/dsl.py

+#===----------------------------------------------------------------------===##
+#
+# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+# See https://llvm.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+#===----------------------------------------------------------------------===##
+
+import distutils.util
+import lit
+import os
+import subprocess
+
+def _memoize(f):
+  cache = dict()
+  def memoized(x):
+    if x not in cache:
+      cache[x] = f(x)
+    return cache[x]
+  return memoized
+
+@_memoize
+def _subprocess_call(command):
+  devNull = open(os.devnull, 'w')
+  return subprocess.call(command, shell=True, stdout=devNull, stderr=devNull)
+
+@_memoize
+def _subprocess_check_output(command):
+  devNull = open(os.devnull, 'w')
+  return subprocess.check_output(command, shell=True, stderr=devNull)
+
+def _expandSubstitutions(config, command):
+  dummySuite = lit.Test.TestSuite('configuration', config.test_source_root, config.test_exec_root, config)
+  dummyTest = lit.Test.Test(dummySuite, ['dummy'], config)
+  tmpDir, tmpBase = lit.TestRunner.getTempPaths(dummyTest)
+  substitutions = lit.TestRunner.getDefaultSubstitutions(dummyTest, tmpDir, tmpBase,
+                                                         normalize_slashes=True)
+  substitutions.append(('%{file_dependencies}', ''))
+  command = lit.TestRunner.applySubstitutions([command], substitutions,
+                                              recursion_limit=config.recursiveExpansionLimit)[0]
+  return command
+
+
+def hasCompileFlag(config, flag):
+  """
+  Return whether the compiler in the configuration supports a given compiler flag.
+
+  This is done by executing the %{cxx} substitution with the given flag and
+  checking whether that succeeds.
+  """
+  command = "echo | %{{cxx}} -xc++ -Werror -fsyntax-only %{{compile_flags}} {} -".format(flag)
+  command = _expandSubstitutions(config, command)
+  result = _subprocess_call(command)
+  return result == 0
+
+def hasLocale(config, locale):
+  """
+  Return whether the runtime execution environment supports a given locale.
+
+  This is done by executing a command to check which locales are available
+  using %{exec} -- this means that the command may be executed on a remote
+  host depending on the %{exec} substitution.
+
+  TODO:
+  - Port to Windows, which doesn't support `locale -a`.
+  - How would using %{exec} during configuration play with deferred execution?
+  """
+  command = _expandSubstitutions(config, "%{exec} locale -a")
+  output = _subprocess_check_output(command).strip()
+  return locale in output
+
+def compilerMacros(config, flags=''):
+  """
+  Return a dictionary of predefined compiler macros.
+
+  The keys are strings representing macros, and the values are strings
+  representing what each macro is defined to.
+
+  If the optional `flags` argument (a string) is provided, these flags will
+  be added to the compiler invocation when generating the macros.
+  """
+  command = "echo | %{{cxx}} -xc++ -dM -E %{{compile_flags}} {} -".format(flags)
+  command = _expandSubstitutions(config, command)
+  unparsed = _subprocess_check_output(command)
+  parsedMacros = dict()
+  for line in filter(None, map(str.strip, unparsed.split('\n'))):
+    assert line.startswith('#define ')
+    line = line[len('#define '):]
+    macro, _, value = line.partition(' ')
+    parsedMacros[macro] = value
+  return parsedMacros
+
+def featureTestMacros(config, flags=''):
+  """
+  Return a dictionary of feature test macros.
+
+  The keys are strings representing feature test macros, and the values are
+  integers representing the value of the macro.
+  """
+  allMacros = compilerMacros(config, flags)
+  return {m: int(v.rstrip('LlUu')) for (m, v) in allMacros.items() if m.startswith('__cpp_')}
+
+
+class Feature(object):
+  """
+  Represents a Lit available feature that is enabled whenever it is supported.
+
+  A feature like this informs the test suite about a capability of the compiler,
+  platform, etc. Unlike Parameters, it does not make sense to explicitly
+  control whether a Feature is enabled -- it should be enabled whenever it
+  is supported.
+  """
+  def __init__(self, name, compileFlag=None, linkFlag=None, when=lambda _: True):
+    """
+    Create a Lit feature for consumption by a test suite.
+
+    - name
+        The name of the feature. This is what will end up in Lit's available
+        features if the feature is enabled. This can be either a string or a
+        callable, in which case it is passed the TestingConfig and should
+        generate a string representing the name of the feature.
+
+    - compileFlag
+        An optional compile flag to add when this feature is added to a
+        TestingConfig. If provided, this must be a string representing a
+        compile flag that will be appended to the end of the %{compile_flags}
+        substitution of the TestingConfig.
+
+    - linkFlag
+        An optional link flag to add when this feature is added to a
+        TestingConfig. If provided, this must be a string representing a
+        link flag that will be appended to the end of the %{link_flags}
+        substitution of the TestingConfig.
+
+    - when
+        A callable that gets passed a TestingConfig and should return a
+        boolean representing whether the feature is supported in that
+        configuration. For example, this can use `hasCompileFlag` to
+        check whether the compiler supports the flag that the feature
+        represents. If omitted, the feature will always be considered
+        supported.
+    """
+    self._name = name
+    self._compileFlag = compileFlag
+    self._linkFlag = linkFlag
+    self._isSupported = when
+
+  def isSupported(self, config):
+    """
+    Return whether the feature is supported by the given TestingConfig.
+    """
+    return self._isSupported(config)
+
+  def enableIn(self, config):
+    """
+    Enable a feature in a TestingConfig.
+
+    The name of the feature is added to the set of available features of
+    `config`, and any compile or link flags provided upon construction of
+    the Feature are added to the end of the corresponding substitution in
+    the config.
+
+    It is an error to call `f.enableIn(cfg)` if the feature `f` is not
+    supported in that TestingConfig (i.e. if `not f.isSupported(cfg)`).
+    """
+    assert self.isSupported(config), \
+      "Trying to enable feature {} that is not supported in the given configuration".format(self._name)
+
+    addTo = lambda subs, sub, flag: [(s, x + ' ' + flag) if s == sub else (s, x) for (s, x) in subs]
+    if self._compileFlag:
+      config.substitutions = addTo(config.substitutions, '%{compile_flags}', self._compileFlag)
+    if self._linkFlag:
+      config.substitutions = addTo(config.substitutions, '%{link_flags}', self._linkFlag)
+
+    name = self._name(config) if callable(self._name) else self._name
+    config.available_features.add(name)
+
+
+class Parameter(object):
+  """
+  Represents a parameter of a Lit test suite.
+
+  Parameters are used to customize the behavior of test suites in a user
+  controllable way, more specifically by passing `--param <KEY>=<VALUE>`
+  when running Lit. Parameters have multiple possible values, and they can
+  have a default value when left unspecified.
+
+  Parameters can have a Feature associated to them, in which case the Feature
+  is added to the TestingConfig if the parameter is enabled. It is an error if
+  the Parameter is enabled but the Feature associated to it is not supported,
+  for example trying to set the compilation standard to C++17 when `-std=c++17`
+  is not supported by the compiler.
+
+  One important point is that Parameters customize the behavior of the test
+  suite in a bounded way, i.e. there should be a finite set of possible choices
+  for `<VALUE>`. While this may appear to be an aggressive restriction, this
+  is actually a very important constraint that ensures that the set of
+  configurations supported by a test suite is finite. Otherwise, a test
+  suite could have an unbounded number of supported configurations, and
+  nobody wants to be stuck maintaining that. If it's not possible for an
+  option to have a finite set of possible values (e.g. the path to the
+  compiler), it can be handled in the `lit.cfg`, but it shouldn't be
+  represented with a Parameter.
+  """
+  def __init__(self, name, choices, type, help, feature, default=None):
+    """
+    Create a Lit parameter to customize the behavior of a test suite.
+
+    - name
+        The name of the parameter that can be used to set it on the command-line.
+        On the command-line, the parameter can be set using `--param <name>=<value>`
+        when running Lit. This must be non-empty.
+
+    - choices
+        A non-empty set of possible values for this parameter. This must be
+        anything that can be iterated. It is an error if the parameter is
+        given a value that is not in that set, whether explicitly or through
+        a default value.
+
+    - type
+        A callable that can be used to parse the value of the parameter given
+        on the command-line. As a special case, using the type `bool` also
+        allows parsing strings with boolean-like contents.
+
+    - help
+        A string explaining the parameter, for documentation purposes.
+        TODO: We should be able to surface those from the Lit command-line.
+
+    - feature
+        A callable that gets passed the parsed value of the parameter (either
+        the one passed on the command-line or the default one), and that returns
+        either None or a Feature.
+
+    - default
+        An optional default value to use for the parameter when no value is
+        provided on the command-line. If the default value is a callable, it
+        is called with the TestingConfig and should return the default value
+        for the parameter. Whether the default value is computed or specified
+        directly, it must be in the 'choices' provided for that Parameter.
+    """
+    self._name = name
+    if len(self._name) == 0:
+      raise ValueError("Parameter name must not be the empty string")
+
+    self._choices = list(choices) # should be finite
+    if len(self._choices) == 0:
+      raise ValueError("Parameter '{}' must be given at least one possible value".format(self._name))
+
+    self._parse = lambda x: (distutils.util.strtobool(x) if type is bool and isinstance(x, str)
+                                                         else type(x))
+    self._help = help
+    self._feature = feature
+    self._default = default
+
+  @property
+  def name(self):
+    """
+    Return the name of the parameter.
+
+    This is the name that can be used to set the parameter on the command-line
+    when running Lit.
+    """
+    return self._name
+
+  def getFeature(self, config, litParams):
+    param = litParams.get(self.name, None)
+    if param is None and self._default is None:
+      raise ValueError("Parameter {} doesn't have a default value, but it was not specified in the Lit parameters".format(self.name))
+    getDefault = lambda: self._default(config) if callable(self._default) else self._default
+    value = self._parse(param) if param is not None else getDefault()
+    if value not in self._choices:
+      raise ValueError("Got value '{}' for parameter '{}', which is not in the provided set of possible choices: {}".format(value, self.name, self._choices))
+    return self._feature(value)