Add remote testing support to the lit config.

[jroelofs]

Summary

This doesn't claim to support ShTest tests yet, but I've added in a hacked-together wrapper script remoteRun.py as one possible solution. I'm still not certain how to orchestrate executable copy-in for those cases.

Here I've also punted on what to do about test data files. The affected tests are:

libc++ :: std/input.output/file.streams/fstreams/filebuf.virtuals/pbackfail.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/filebuf.virtuals/underflow.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.assign/member_swap.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.assign/move_assign.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.assign/nonmember_swap.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.cons/move.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.cons/pointer.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.cons/string.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.members/close.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.members/open_pointer.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.members/open_string.pass.cpp
libc++ :: std/input.output/file.streams/fstreams/ifstream.members/rdbuf.pass.cpp
libc++ :: std/localization/locales/locale.convenience/conversions/conversions.buffer/pbackfail.pass.cpp
libc++ :: std/localization/locales/locale.convenience/conversions/conversions.buffer/underflow.pass.cpp

I tested this on my mac with an SSHExecutor that connects to localhost, and copies executables into /tmp to get a feel for the overhead. That run took ~414s, whereas a normal one takes about half that at ~204s.

This isn't quite ready for commit, but it's a lot closer... There's still a few things I'd like to pick your brains on, @danalbert and @EricWF.

[jroelofs]

remoteRun.py

8 ↗

(On Diff #19239)

I considered passing this in as the first argument, and using eval. Better ideas?

[jroelofs]

libcxx/test/config.py

124 ↗

(On Diff #19239)

I forgot a

else:
    self.target_executor = None

here

libcxx/test/format.py

121 ↗

(On Diff #19239)

I kept these cases separate to avoid clashing with your configuration, @danalbert. It might be better to drop the else side of this, and have target_executor be a LocalExecutor instead of None when doing local testing. I'll save that decision for later to minimize invasiveness.

libcxx/test/remote.py

6 ↗

(On Diff #19239)

I couldn't figure out how to get the include of lit.util to work here when using the wrapper script. It would be better to include that than to duplicate the code here.

lit.site.cfg.in

21 ↗

(On Diff #19239)

This part is obviously not supposed to go upstream, but I put it in the patch to show how to use it.

I think there are at least two common patterns that might be worth exposing through cmake:

config.target_executor = "SSHExecutor('username', 'hostname')"

config.target_executor = "PrefixExecutor('/path/to/run/script', LocalExecutor())"

Thoughts? Should we just pass the whole blob in on the cmake configure command line, or do something more sophisticated?

[jroelofs]

Fix the LocalExecutor

[jroelofs]

Set target_executor to None when there isn't one.

[jroelofs]

Another thing I haven't attempted to deal with in this patch is that there are a lot of assumptions that $host==$target for test parameter discovery. Need to clean those up at a later point.

test/libcxx/test/config.py

290

I don't know where this came from. Bad merge maybe?

test/libcxx/test/remote.py

203

This one is already deprecated. Going to remove it as soon as the lit timeouts patch goes in.

319

This class screams that it wants reflection... I just don't know how to do that.

test/lit.site.cfg.in

21

Again, I don't want to upstream this part... it's just an example.

test/remoteRun.py

1 ↗

(On Diff #19418)

This file isn't used yet, but we could use it (or something like it) to to ShTest tests.

[danalbert]

I'm happy this is mostly unintrusive. I was expecting a bigger mess :)

If you decide not to go with my suggestions on L122, I'll probably have more to say about the executors themselves, otherwise I'll wait for the next revision since that will completely change them.

libcxx/test/format.py

98 ↗

(On Diff #19239)

No reason for this to be an argument, it's a member.

121 ↗

(On Diff #19239)

and have target_executor be a LocalExecutor instead of None when doing local testing

Was going to suggest exactly that. Should keep things cleaner and make sure neither side bit rots (because there will only be one side).

test/libcxx/test/config.py

15

Should avoid wildcard imports.

122

Ewwwwwww eval? Unless there's a really good reason to use an expression that can be evaluated over just naming a type, I think this should be:

mod_path, _, executor = exec_str.rpartition('.')
mod = importlib.import_module(mod_path)
self.target_executor = getattr(mod, executor)()

Where exec_str is the fully qualified type name (i.e. libcxx.test.remote.LocalExecutor). Then the wildcard import and the ugly eval can go away.

290

Yes. ericwf added this the other day because it was something that went missing from the libcxxabi config (one test has a // REQUIRES: linux2)

test/libcxx/test/format.py

119

I think this part of the logic could probably just be pushed down in to the executor. The parameters to exc.run() would become exec_path, source_path, env={}, valgrind_mess. exc.copy_in() becomes unnecessary, be the executor can decide for itself what should be copied. Same goes for exc.remote_from_local_dir() and exc.remote_from_local_file(), and exc.delete_remote(). This entire added block (and some the the existing code above) would become:

out, err, rc = executor.run(exec_path, source_path, self.exec_env, valgrind_mess)

That should also help with replacing target_executor = None with executor = LocalExecutor(), as each method of LocalExecutor would become a no-op.

This will probably cause some duplication between the various remote executors, so it might make sense to have a common base for all of them that factors that out.

131

This should stay as it was. The target executor should clean up its own mess, but not the mess from the higher level (this is true with and without my suggestion at L122).

test/libcxx/test/remote.py

7

What makes import hard here?

194

This would be a nice way to factor out that cruft. Good thinking.

213

Don't align assignments (you haven't matched all of them anyway).

224

Your naming is inconsistent. self.scpCommand but self.remote_temp_files. I prefer PEP8 stlye (with underscores), but LIT already chose the anti-Python style, so let's try to stick with that. (I'm guilty of this crime too.)

256

if local_file not in self.filemap:

307

Use != for comparing strings rather than is not. is not should be used to test that two things are exactly the same object.

319

You probably want something along the lines of http://stackoverflow.com/a/3467879/632035

Should make the whole class unnecessary.

[jroelofs]

test/libcxx/test/format.py

131

Good point.

test/libcxx/test/remote.py

7

when imported from remoteRun.py, the imports here don't work because we do funny business somewhere in the lit.cfg that adds the llvm imports to some path that python uses to search from.

I'll add the imports, drop this duped code, and drop the remoteRun.py. Can add that back and fix the import problem later when we try to get ShTest going.

213

ok.

I really wish phabricator used a monospaced font for diffs.... haven't been able to find a setting that changes that.

224

as in camelCase for vars, and underscore_separated_functions() ? Ok.

[danalbert]

test/libcxx/test/remote.py

224

Actually, looking at lit's Test.py, it looks like the format is underscores for variables, camel-case for functions.

[jroelofs]

Address most of @danalbert's comments. There are still a few remaining that I need to take care of though.

[jroelofs]

test/libcxx/test/config.py

122

This means that the executors cannot be configured or chained via their constructor arguments, which is something I think is necessary to make, say, SSHExecutors work.... unless I start parsing the args, which leads to just re-inventing eval.

124

Drop this line.

test/libcxx/test/format.py

119

Still need to push most of this into target_executor.run

test/libcxx/test/remote.py

116

!=

284

I think this is much better, but it's also extremely complicated now. :/

[danalbert]

test/libcxx/test/config.py

122

Ah, right. That makes sense. Carry on.

[jroelofs]

Address the rest of @danalbert's comments

[jroelofs]

Add TargetInfo class for determining things like locale support and platform name.

[jroelofs]

Uploaded a diff from the wrong dir. Sorry for the spam.

[jroelofs]

Remove something I was using for testing an idea. It didn't pan out.

[jroelofs]

Make capitalization consistent.

[EricWF]

Drive by comments.

test/libcxx/test/config.py

120

Isn't this always set to the default by CMake?

122

Remove this. Use the lit_config reporting methods.

test/libcxx/test/format.py

78

Isn't this always true? Can we detect wether we have a local executor for the time being?

test/libcxx/test/remote.py

11–21

I would throw a NotImplemented or similar error from these "abstract" methods

52

I would probably throw from these abstract methods too if they shouldn't be called.

[jroelofs]

Address @EricWF's driveby comments.

[danalbert]

Looking better. At first glance, I think there's some more cleanup that can be done in remote.py, but I'll finish the review first thing tomorrow.

BTW, I like the TargetInfo bit. It looks like that change is actually cleanly separates from the rest of this. That part LGTM, so go ahead and submit that portion as its own commit.

test/libcxx/test/config.py

2

Unused.

116

Now that even the base case is a LocalExecutor, I think the target_ prefix of this is misleading.

120

Default should be "LocalExecutor". Then the if/else can disappear.

128

useValgrind should just go away. If the user wants valgrind, they can add the ValgrindExecutor themselves.

599–600

Use parens rather than the \

And keep the not with the thing it's negating.

test/libcxx/test/format.py

89

No reason to pass this as an argument. Just refer to it as self.target_executor in _evaluate_pass_test.

118–119

The above three lines are just env = self.exec_env.

118–119

Ignore the above. I traced it back and found that the default self.exec_env is {}, not None. Phabricator isn't letting me delete the comment though...

[danalbert]

Here's my take on remote.py: https://gist.github.com/DanAlbert/a45ac580c3fc61917856

This patch no longer applies cleanly to ToT, and I don't know what revision it was generate against, so I wasn't able to apply this to my working tree to test it. Once it's rebased I'll verify.

[jroelofs]

Rebase & use @danalbert's remote.py + tracing.py almost verbatim.

[jroelofs]

Address (what I think are) the rest of @danalbert's comments on this patch.

[jroelofs]

I split the TargetInfo part into its own patch: http://reviews.llvm.org/D7601

This patch will need a slight rebase after that goes in.

libcxx/test/config.py

471 ↗

(On Diff #19631)

I think this needs to stay as lowercase, actually.

test/libcxx/test/config.py

120

Oh yeah.

122

oops

128

This suggestion is a buildbot-breaking change. Can clean this up after any zorg bots that care about valgrind are moved over to using the new cmake flag.

[jroelofs]

Fix the race condition in SSHExecutor, and leave a note saying to be careful.

[jroelofs]

Didn't get all the files in the diff. Sorry for the spam.

[jroelofs]

libcxx/test/config.py

608 ↗

(On Diff #19915)

I think this is evidence that I need to rebase the patch. The target_info stuff already went in.

libcxx/test/tracing.py

6 ↗

(On Diff #19915)

delete this line.

[danalbert]

LGTM with the one nit.

Thanks a lot for taking care of this. Should greatly simplify my test setup.

libcxx/test/config.py

125 ↗

(On Diff #19915)

This isn't inferred, this is explicit. The else path is inferred.

[jroelofs]

And with much rejoicing, there was r230592.

test/CMakeLists.txt

set(LIBCXX_LIT_VARIANT "libcxx" CACHE STRING
       "Configuration variant to use for LIT.")
 
   pythonize_bool(LIBCXX_ENABLE_EXCEPTIONS)
   pythonize_bool(LIBCXX_ENABLE_RTTI)
   pythonize_bool(LIBCXX_ENABLE_SHARED)
   pythonize_bool(LIBCXX_BUILD_32_BITS)
   pythonize_bool(LIBCXX_ENABLE_THREADS)
   pythonize_bool(LIBCXX_ENABLE_MONOTONIC_CLOCK)
+  set(LIBCXX_TARGET_EXECUTOR "LocalExecutor()" CACHE STRING
+      "Executor to use when running tests.")
 
   set(AUTO_GEN_COMMENT "## Autogenerated by libcxx configuration.\n# Do not edit!")
 
   configure_file(
     ${CMAKE_CURRENT_SOURCE_DIR}/lit.site.cfg.in
     ${CMAKE_CURRENT_BINARY_DIR}/lit.site.cfg
     @ONLY)
 

test/libcxx/test/config.py

+import importlib
 import locale
 import os
 import platform
 import pkgutil
 import re
 import shlex
 import sys
 
 import lit.Test  # pylint: disable=import-error,no-name-in-module
 import lit.util  # pylint: disable=import-error,no-name-in-module
 
 from libcxx.test.format import LibcxxTestFormat
 from libcxx.compiler import CXXCompiler
-
+from libcxx.test.remote import *
 
 def loadSiteConfig(lit_config, config, param_name, env_name):
     # We haven't loaded the site specific configuration (the user is
     # probably trying to run on a test file directly, and either the site
     # configuration hasn't been created by the build system, or we are in an
     # out-of-tree build situation).
     site_cfg = lit_config.params.get(param_name,
                                      os.environ.get(env_name))
def __init__(self, lit_config, config):
         self.cxx_library_root = None
         self.env = {}
         self.use_target = False
         self.use_system_cxx_lib = False
         self.use_clang_verify = False
         self.long_tests = None
         self.execute_external = False
 
-        if platform.system() not in ('Darwin', 'FreeBSD', 'Linux'):
-            self.lit_config.fatal("unrecognized system")
-
     def get_lit_conf(self, name, default=None):
         val = self.lit_config.params.get(name, None)
         if val is None:
             val = getattr(self.config, name, None)
             if val is None:
                 val = default
         return val
 
     def get_lit_bool(self, name, default=None):
         conf = self.get_lit_conf(name)
         if conf is None:
             return default
         if conf.lower() in ('1', 'true'):
             return True
         if conf.lower() in ('', '0', 'false'):
             return False
         self.lit_config.fatal(
             "parameter '{}' should be true or false".format(name))
 
     def configure(self):
+        self.configure_target_executor()
+        self.configure_target_info()
         self.configure_cxx()
         self.configure_triple()
         self.configure_src_root()
         self.configure_obj_root()
         self.configure_cxx_library_root()
         self.configure_use_system_cxx_lib()
         self.configure_use_clang_verify()
         self.configure_execute_external()
def print_config_info(self):
                              list(self.config.available_features))
         self.lit_config.note('Using environment: %r' % self.env)
 
     def get_test_format(self):
         return LibcxxTestFormat(
             self.cxx,
             self.use_clang_verify,
             self.execute_external,
+            self.target_executor,
             exec_env=self.env)
 
+    def configure_target_executor(self):
+        exec_str = self.get_lit_conf('target_executor', None)
+        if exec_str:
+            print exec_str
+            self.target_executor = eval(exec_str)
+            self.lit_config.note("inferred target_executor as: %r" % exec_str)
+            if self.lit_config.useValgrind:
+                # We have no way of knowing where in the chain the
+                # ValgrindExecutor is supposed to go. It is likely
+                # that the user wants it at the end, but we have no
+                # way of getting at that easily.
+                selt.lit_config.fatal("Cannot infer how to create a Valgrind "
+                                      " target_executor.")
+        else:
+            te = LocalExecutor()
+            if self.lit_config.useValgrind:
+                te = ValgrindExecutor(self.lit_config.valgrindArgs, te)
+            self.target_executor = te
+
+    def configure_target_info(self):
+        info_str = self.get_lit_conf('target_info', None)
+        if info_str:
+            self.target_info = eval(info_str)
+            self.lit_config.note("inferred target_info as: %r" % info_str)
+        else:
+            self.target_info = LocalTI()
+
     def configure_cxx(self):
         # Gather various compiler parameters.
         cxx = self.get_lit_conf('cxx_under_test')
 
         # If no specific cxx_under_test was given, attempt to infer it as
         # clang++.
         if cxx is None:
             clangxx = lit.util.which('clang++',
class Configuration(object):
     def configure_features(self):
         additional_features = self.get_lit_conf('additional_features')
         if additional_features:
             for f in additional_features.split(','):
                 self.config.available_features.add(f.strip())
 
         # Figure out which of the required locales we support
         locales = {
-            'Darwin': {
+            'darwin': {
                 'en_US.UTF-8': 'en_US.UTF-8',
                 'cs_CZ.ISO8859-2': 'cs_CZ.ISO8859-2',
                 'fr_FR.UTF-8': 'fr_FR.UTF-8',
                 'fr_CA.ISO8859-1': 'cs_CZ.ISO8859-1',
                 'ru_RU.UTF-8': 'ru_RU.UTF-8',
                 'zh_CN.UTF-8': 'zh_CN.UTF-8',
             },
-            'FreeBSD': {
+            'freebsd': {
                 'en_US.UTF-8': 'en_US.UTF-8',
                 'cs_CZ.ISO8859-2': 'cs_CZ.ISO8859-2',
                 'fr_FR.UTF-8': 'fr_FR.UTF-8',
                 'fr_CA.ISO8859-1': 'fr_CA.ISO8859-1',
                 'ru_RU.UTF-8': 'ru_RU.UTF-8',
                 'zh_CN.UTF-8': 'zh_CN.UTF-8',
             },
-            'Linux': {
+            'linux': {
                 'en_US.UTF-8': 'en_US.UTF-8',
                 'cs_CZ.ISO8859-2': 'cs_CZ.ISO-8859-2',
                 'fr_FR.UTF-8': 'fr_FR.UTF-8',
                 'fr_CA.ISO8859-1': 'fr_CA.ISO-8859-1',
                 'ru_RU.UTF-8': 'ru_RU.UTF-8',
                 'zh_CN.UTF-8': 'zh_CN.UTF-8',
             },
-            'Windows': {
+            'windows': {
                 'en_US.UTF-8': 'English_United States.1252',
                 'cs_CZ.ISO8859-2': 'Czech_Czech Republic.1250',
                 'fr_FR.UTF-8': 'French_France.1252',
                 'fr_CA.ISO8859-1': 'French_Canada.1252',
                 'ru_RU.UTF-8': 'Russian_Russia.1251',
                 'zh_CN.UTF-8': 'Chinese_China.936',
             },
         }
 
+        target_platform = self.target_info.platform()
+
         default_locale = locale.setlocale(locale.LC_ALL)
-        for feature, loc in locales[platform.system()].items():
-            try:
-                locale.setlocale(locale.LC_ALL, loc)
+        if target_platform in locales:
+            for feature, loc in locales[target_platform].items():
+                if self.target_info.supports_locale(loc):
                     self.config.available_features.add(
                         'locale.{0}'.format(feature))
-            except locale.Error:
-                self.lit_config.warning('The locale {0} is not supported by '
-                                        'your platform. Some tests will be '
-                                        'unsupported.'.format(loc))
+                else:
+                    self.lit_config.warning('The locale {0} is not supported by'
+                                            ' your platform. Some tests will be'
+                                            ' unsupported.'.format(loc))
         locale.setlocale(locale.LC_ALL, default_locale)
 
         # Write an "available feature" that combines the triple when
         # use_system_cxx_lib is enabled. This is so that we can easily write
         # XFAIL markers for tests that are known to fail with versions of
         # libc++ as were shipped with a particular triple.
         if self.use_system_cxx_lib:
             self.config.available_features.add(
                 'with_system_cxx_lib=%s' % self.config.target_triple)
 
+        # Insert the platform name into the available features as a lower case.
+        # Strip the '2' from linux2.
+        if target_platform.startswith('linux'):
+            platform_name = 'linux'
+        else:
+            platform_name = target_platform
+        self.config.available_features.add(platform_name.lower())
+
         # Some linux distributions have different locale data than others.
         # Insert the distributions name and name-version into the available
         # features to allow tests to XFAIL on them.
-        if sys.platform.startswith('linux'):
-            name, ver, _ = platform.linux_distribution()
-            name = name.lower().strip()
-            ver = ver.lower().strip()
+        if target_platform.startswith('linux'):
+            name = self.target_info.platform_name()
+            ver = self.target_info.platform_ver()
             if name:
                 self.config.available_features.add(name)
             if name and ver:
                 self.config.available_features.add('%s-%s' % (name, ver))
 
         # Simulator testing can take a really long time for some of these tests
         # so add a feature check so we can REQUIRES: long_tests in them
         self.long_tests = self.get_lit_bool('long_tests')
def configure_default_compile_flags(self):
         # default given in lit.site.cfg is not present. If default is not
         # present then force c++11.
         std = self.get_lit_conf('std', 'c++11')
         self.cxx.compile_flags += ['-std={0}'.format(std)]
         self.config.available_features.add(std)
         # Configure include paths
         self.cxx.compile_flags += ['-nostdinc++']
         self.configure_compile_flags_header_includes()
-        if sys.platform.startswith('linux'):
+        if self.target_info.platform().startswith('linux'):
             self.cxx.compile_flags += ['-D__STDC_FORMAT_MACROS',
                                        '-D__STDC_LIMIT_MACROS',
                                        '-D__STDC_CONSTANT_MACROS']
         # Configure feature flags.
         self.configure_compile_flags_exceptions()
         self.configure_compile_flags_rtti()
         enable_32bit = self.get_lit_bool('enable_32bit', False)
         if enable_32bit:
def configure_link_flags_abi_library(self):
             pass
         else:
             self.lit_config.fatal(
                 'C++ ABI setting %s unsupported for tests' % cxx_abi)
 
     def configure_extra_library_flags(self):
         enable_threads = self.get_lit_bool('enable_threads', True)
         llvm_unwinder = self.get_lit_bool('llvm_unwinder', False)
-        if sys.platform == 'darwin':
+        target_platform = self.target_info.platform()
+        if target_platform == 'darwin':
             self.cxx.link_flags += ['-lSystem']
-        elif sys.platform.startswith('linux'):
+        elif target_platform.startswith('linux'):
             if not llvm_unwinder:
                 self.cxx.link_flags += ['-lgcc_eh']
             self.cxx.link_flags += ['-lc', '-lm']
             if enable_threads:
                 self.cxx.link_flags += ['-lpthread']
+            if not target_platform.startswith('baremetal'):
                 self.cxx.link_flags += ['-lrt']
             if llvm_unwinder:
                 self.cxx.link_flags += ['-lunwind', '-ldl']
             else:
                 self.cxx.link_flags += ['-lgcc_s']
-        elif sys.platform.startswith('freebsd'):
+        elif target_platform.startswith('freebsd'):
             self.cxx.link_flags += ['-lc', '-lm', '-lpthread', '-lgcc_s']
         else:
-            self.lit_config.fatal("unrecognized system: %r" % sys.platform)
+            self.lit_config.fatal("unrecognized system: %r" %
+                                  target_platform)
 
     def configure_sanitizer(self):
         san = self.get_lit_conf('use_sanitizer', '').strip()
         if san:
             # Search for llvm-symbolizer along the compiler path first
             # and then along the PATH env variable.
             symbolizer_search_paths = os.environ.get('PATH', '')
             cxx_path = lit.util.which(self.cxx.path)
             if cxx_path is not None:
                 symbolizer_search_paths = (
                     os.path.dirname(cxx_path) +
                     os.pathsep + symbolizer_search_paths)
             llvm_symbolizer = lit.util.which('llvm-symbolizer',
                                              symbolizer_search_paths)
             # Setup the sanitizer compile flags
             self.cxx.flags += ['-g', '-fno-omit-frame-pointer']
-            if sys.platform.startswith('linux'):
+            if self.target_info.platform().startswith('linux'):
                 self.cxx.link_flags += ['-ldl']
             if san == 'Address':
                 self.cxx.flags += ['-fsanitize=address']
                 if llvm_symbolizer is not None:
                     self.env['ASAN_SYMBOLIZER_PATH'] = llvm_symbolizer
                 self.config.available_features.add('asan')
             elif san == 'Memory' or san == 'MemoryWithOrigins':
                 self.cxx.flags += ['-fsanitize=memory']
def configure_triple(self):
             if target_triple.endswith('redhat-linux') or \
                target_triple.endswith('suse-linux'):
                 target_triple += '-gnu'
             self.config.target_triple = target_triple
             self.lit_config.note(
                 "inferred target_triple as: %r" % self.config.target_triple)
 
     def configure_env(self):
-        if sys.platform == 'darwin' and not self.use_system_cxx_lib:
+        if self.target_info.platform() == 'darwin' and not \
+           self.use_system_cxx_lib:
             libcxx_library = self.get_lit_conf('libcxx_library')
             if libcxx_library:
                 cxx_library_root = os.path.dirname(libcxx_library)
             else:
                 cxx_library_root = self.cxx_library_root
             self.env['DYLD_LIBRARY_PATH'] = cxx_library_root

test/libcxx/test/format.py

class LibcxxTestFormat(object):
 
     Tests fall into two categories:
       FOO.pass.cpp - Executable test which should compile, run, and exit with
                      code 0.
       FOO.fail.cpp - Negative test case which is expected to fail compilation.
       FOO.sh.cpp   - A test that uses LIT's ShTest format.
     """
 
-    def __init__(self, cxx, use_verify_for_fail, execute_external, exec_env):
+    def __init__(self, cxx, use_verify_for_fail, execute_external,
+                 target_executor, exec_env):
         self.cxx = cxx
         self.use_verify_for_fail = use_verify_for_fail
         self.execute_external = execute_external
+        self.target_executor = target_executor
         self.exec_env = dict(exec_env)
 
     # TODO: Move this into lit's FileBasedTest
     def getTestsInDirectory(self, testSuite, path_in_suite,
                             litConfig, localConfig):
         source_path = testSuite.getSourcePath(path_in_suite)
         for filename in os.listdir(source_path):
             # Ignore dot files and excluded tests.
def _execute(self, test, lit_config):
         # res is not an instance of lit.test.Result. Expand res into its parts.
         script, tmpBase, execDir = res
         # Check that we don't have run lines on tests that don't support them.
         if not is_sh_test and len(script) != 0:
             lit_config.fatal('Unsupported RUN line found in test %s' % name)
 
         # Dispatch the test based on its suffix.
         if is_sh_test:
+            if self.target_executor:
+                # We can't run ShTest tests with a target_executor yet.
+                # For now, bail on trying to run them
+                return lit.Test.UNSUPPORTED, 'ShTest format not yet supported'
             return lit.TestRunner._runShTest(test, lit_config,
                                              self.execute_external, script,
                                              tmpBase, execDir)
         elif is_fail_test:
             return self._evaluate_fail_test(test)
         elif is_pass_test:
-            return self._evaluate_pass_test(test, tmpBase, execDir, lit_config)
+            return self._evaluate_pass_test(test, tmpBase, execDir, lit_config,
+                                            self.target_executor)
         else:
             # No other test type is supported
             assert False
 
     def _clean(self, exec_path):  # pylint: disable=no-self-use
         libcxx.util.cleanFile(exec_path)
 
-    def _evaluate_pass_test(self, test, tmpBase, execDir, lit_config):
+    def _evaluate_pass_test(self, test, tmpBase, execDir, lit_config,
+                            target_executor):
         source_path = test.getSourcePath()
         exec_path = tmpBase + '.exe'
         object_path = tmpBase + '.o'
         # Create the output directory if it does not already exist.
         lit.util.mkdir_p(os.path.dirname(tmpBase))
         try:
             # Compile the test
             cmd, out, err, rc = self.cxx.compileLinkTwoSteps(
                 source_path, out=exec_path, object_file=object_path,
                 cwd=execDir)
             compile_cmd = cmd
             if rc != 0:
                 report = libcxx.util.makeReport(cmd, out, err, rc)
                 report += "Compilation failed unexpectedly!"
                 return lit.Test.FAIL, report
             # Run the test
-            cmd = []
+            local_cwd = os.path.dirname(source_path)
+            env = None
             if self.exec_env:
-                cmd += ['env']
-                cmd += ['%s=%s' % (k, v) for k, v in self.exec_env.items()]
-            if lit_config.useValgrind:
-                cmd = lit_config.valgrindArgs + cmd
-            cmd += [exec_path]
-            out, err, rc = lit.util.executeCommand(
-                cmd, cwd=os.path.dirname(source_path))
+                env = self.exec_env
+            out, err, rc = target_executor.copy_run(exec_path, [], local_cwd, env)
             if rc != 0:
                 report = libcxx.util.makeReport(cmd, out, err, rc)
                 report = "Compiled With: %s\n%s" % (compile_cmd, report)
                 report += "Compiled test failed unexpectedly!"
                 return lit.Test.FAIL, report
             return lit.Test.PASS, ''
         finally:
             # Note that cleanup of exec_file happens in `_clean()`. If you
             # override this, cleanup is your reponsibility.
             libcxx.util.cleanFile(object_path)
             self._clean(exec_path)
 
     def _evaluate_fail_test(self, test):
         source_path = test.getSourcePath()
         # TODO: Move the checking of USE_VERIFY into
         # lit.TestRunner.parseIntegratedTestScript by adding support for custom
         # tags.
         with open(source_path, 'r') as f:
             contents = f.read()
         use_verify = 'USE_VERIFY' in contents and self.use_verify_for_fail

test/libcxx/test/remote.py

+import inspect
+import locale
+import os
+import platform
+import shutil
+import types
+
+from lit.util import executeCommand
+
+class TargetInfo(object):
+    def platform(self):
+        pass
+
+    def platform_ver(self):
+        pass
+
+    def platform_name(self):
+        pass
+
+    def supports_locale(self, loc):
+        pass
+
+
+class LocalTI(TargetInfo):
+    def platform(self):
+        return platform.system().lower().strip()
+
+    def platform_name(self):
+        if platform().startswith('linux'):
+            name, _, _ = platform.linux_distribution()
+            name = name.lower().strip()
+            if name:
+                return name
+        return None
+
+    def platform_ver(self):
+        if platform().startswith('linux'):
+            _, ver, _ = platform.linux_distribution()
+            ver = ver.lower().strip()
+            if ver:
+                return ver
+        return None
+
+    def supports_locale(self, loc):
+        try:
+            locale.setlocale(locale.LC_ALL, loc)
+            return True
+        except locale.Error:
+            return False
+
+
+class Executor(object):
+    def __init__(self):
+        pass
+
+    def remote_temp_dir(self):
+        """Get a temporary directory on the remote host"""
+        pass
+
+    def remote_temp_file(self, suffix):
+        """Get a temporary filename on the remote host"""
+        pass
+
+    def remote_from_local_dir(self, local_dir):
+        """Translate a local dirname into a remote dirname"""
+        pass
+
+    def remote_from_local_file(self, local_file):
+        """Translate a local filename into a remote filename"""
+        pass
+
+    def copy_in(self, local_srcs, remote_dsts, delete=False):
+        """Copy test inputs & executables in to the test runner
+        Args:
+          local_srcs  (list): List of Host file paths to copy in to the runner
+          remote_dsts (list): Corresponding list of Target file locations
+          delete      (bool): Whether or not to delete the src after the copy
+        Returns:
+          void
+        """
+        pass
+
+    def copy_out(self, remote_srcs, local_dsts, delete=False):
+        """Copy test outputs back from the test runner
+        Args:
+          remote_srcs (list): List of Target file paths to copy back from the runner
+          local_dsts  (list): Corresponding list of Host file locations
+          delete      (bool): Whether or not to delete the src after the copy
+        Returns:
+          void
+        """
+        pass
+
+    def delete_remote(self, remote, ignore_error=True):
+        """Delete remote files
+        Args:
+          remote       (path): Remote file to delete
+          ignore_error (bool): Whether or not to ignore errors on delete
+        Returns:
+          void
+        """
+        pass
+
+    def run(self, command, work_dir='.', env=None):
+        """Execute a particular command
+        Args:
+          command  (string): The command to execute
+          work_dir (string): The working directory to execute the test from
+        Returns:
+          out, err, exitCode
+        """
+        pass
+
+    def copy_run(self, exe_path, args, local_cwd, env=None):
+        """Execute a command that might need to be copied to the target
+        Args:
+          exe_path     (string): Local path to the executable to be run
+          args (list of string): Arguments to run the exe with
+          local_cwd    (string): Local path to the working directory
+          env            (dict): Environment variables to execute under
+        Returns:
+          out, err, exitCode
+        """
+        cmd = args
+        target_exe_path = self.remote_from_local_file(exe_path)
+        target_cwd = self.remote_from_local_dir(local_cwd)
+        cmd += [target_exe_path]
+        try:
+            self.copy_in([exe_path], [target_exe_path])
+            return self.run(cmd, work_dir=target_cwd, env=env)
+        except:
+            raise
+        finally:
+            self.delete_remote(target_exe_path)
+            pass
+
+    def clean_temps(self):
+        """Delete all temporary files created by this executor"""
+        pass
+
+class LocalExecutor(Executor):
+    def __init__(self):
+        Executor.__init__(self)
+
+    def remote_temp_dir(self):
+        return os.getcwd()
+
+    def remote_temp_file(self, suffix):
+        tmp_file = tempfile.NamedTemporaryFile(suffix=suffix, delete=False)
+        tmp_path = tmp_file.name
+        tmp_file.close()
+        return tmp_path
+
+    def delete_remote(self, remote, ignore_error=True):
+        try:
+            os.remove(remote)
+        except:
+            if ignore_error:
+                pass
+            else:
+                raise
+
+    def remote_from_local_dir(self, local_dir):
+        # Identity function, because LocalExecutor executes locally
+        return local_dir
+
+    def remote_from_local_file(self, local_file):
+        # Identity function, because LocalExecutor executes locally
+        return local_file
+
+    def copy_in(self, local_srcs, remote_dsts, delete=False):
+        self.__copyAll(local_srcs, remote_dsts, delete)
+
+    def copy_out(self, remote_srcs, local_dsts, delete=False):
+        self.__copyAll(remote_srcs, local_dsts, delete)
+
+    def run(self, command, work_dir='.', env=None):
+        env_cmd = []
+        if env:
+            env_cmd += ['env']
+            env_cmd += ['%s=%s' % (k, v) for k, v in env.items()]
+        if work_dir == '.':
+            work_dir = os.getcwd()
+        return executeCommand(env_cmd + command, cwd=work_dir)
+
+    def copy_run(self, exe_path, args, local_cwd, env=None):
+        return executeCommand([exe_path] + args, cwd=local_cwd, env=env)
+
+    def __copyAll(self, srcs, dsts, delete=False):
+        for src, dst in zip(srcs, dsts):
+            if src != dst:
+                if delete:
+                    move(src, dst)
+                else:
+                    shutil.copy(src, dst)
+            elif delete:
+                assert false, "Can't delete '%s' because src==dst" % src
+
+class PrefixExecutor(Executor):
+    """Prefix an executor with some other command wrapper
+    Most useful for setting ulimits on commands, or running an emulator like
+    qemu and valgrind.
+    """
+    def __init__(self, commandPrefix, chain):
+        Executor.__init__(self)
+
+        self.commandPrefix = commandPrefix
+        self.chain = chain
+
+    def remote_from_local_dir(self, local_dir):
+        return local_dir
+
+    def remote_from_local_file(self, local_file):
+        return local_file
+
+    def copy_in(self, local_srcs, remote_dsts, delete=False):
+        self.chain.copy_in(local_srcs, remote_dsts, delete)
+
+    def copy_out(self, remote_srcs, local_dsts, delete=False):
+        self.chain.copy_out(remote_srcs, local_dsts, delete)
+
+    def delete_remote(self, remote, ignore_error=True):
+        self.chain.delete_remote(remote, ignore_error)
+
+    def run(self, command, work_dir='.', env=None):
+        return self.chain.run(self.commandPrefix + command, work_dir, env=env)
+
+class PostfixExecutor(PrefixExecutor):
+    """Postfix an executor with some args
+    """
+    def __init__(self, commandPostfix, chain):
+        Executor.__init__(self)
+
+        self.commandPostfix = commandPostfix
+        self.chain = chain
+
+    def run(self, command, work_dir='.', env=None):
+        return self.chain.run(command + self.commandPostfix, work_dir, env=env)
+
+class ValgrindExecutor(PrefixExecutor):
+    pass
+
+class QEMUExecutor(PrefixExecutor):
+    """Execute another action under qemu"""
+    def __init__(self, qemuCommand, qemuArgs, chain):
+        PrefixExecutor.__init__(self, prefix, chain)
+        self.prefix = qemuCommand + ' ' + qemuArgs
+
+# Deprecated. lit timeouts subsume this
+class TimeoutExecutor(PrefixExecutor):
+    """Execute another action under a timeout"""
+    def __init__(self, duration, chain):
+        PrefixExecutor.__init__(self, 'timeout ' + duration, chain)
+
+class SSHExecutor(Executor):
+    def __init__(self, username, host):
+        Executor.__init__(self)
+
+        self.username = username
+        self.host = host
+        self.scp_command = 'scp'
+        self.ssh_command = 'ssh'
+        self.local_executor = LocalExecutor()
+
+        # Maps from local->remote to make sure we consistently return
+        # the same name for a given local version of the same file/dir
+        self.filemap = {}
+        self.dirmap = {}
+
+        # For temp cleanup
+        self.remote_temp_files = []
+        self.remote_temp_dirs = []
+
+        # TODO(jroelofs): switch this on some -super-verbose-debug config flag
+        if False:
+            self.local_executor = DebugExecutor("ssh_local", LocalExecutor(),
+                                                before=True, after=True)
+
+    def remote_temp_dir(self):
+        # TODO
+        return os.getcwd()
+
+    def remote_temp_file(self, suffix):
+        # TODO: detect what the target system is, and use the correct
+        # mktemp command for it. (linux and darwin differ here, and I'm
+        # sure windows has another way to do it)
+
+        # Not sure how to do suffix on osx yet
+        cmd = "mktemp -q /tmp/libcxx.XXXXXXXXXX"
+        res = self.__execute_command_remote([cmd])
+        temp_path, err, exitCode = res
+        temp_path = temp_path.rstrip()
+        if exitCode != 0:
+            raise ValueError(err)
+        self.remote_temp_files.append(temp_path)
+        return temp_path
+
+    def remote_from_local_dir(self, local_dir):
+        if not self.dirmap.has_key(local_dir):
+            remote_dir = self.remote_temp_dir()
+            self.dirmap[local_dir] = remote_dir
+            return remote_dir
+        else:
+            return self.dirmap[local_dir]
+
+    def remote_from_local_file(self, local_file):
+        if not local_file in self.filemap:
+            remote_file = self.remote_temp_file(".some_file")
+            self.filemap[local_file] = remote_file
+            return remote_file
+        else:
+            return self.filemap[local_file]
+
+    def copy_in(self, local_srcs, remote_dsts, delete=False):
+        scp = self.scp_command
+        remote = self.username + '@' + self.host
+
+        # This could be wrapped up in a tar->scp->untar for performance
+        # if there are lots of files to be copied/moved
+        for src, dst in zip(local_srcs, remote_dsts):
+            cmd = [scp, '-p', src, remote + ':' + dst]
+            self.local_executor.run(cmd)
+        if delete:
+            for src in local_srcs:
+                os.remove(src)
+
+    def copy_out(self, remote_srcs, local_dsts, delete=False):
+        scp = self.scp_command
+        remote = self.username + '@' + self.host
+
+        # This could be wrapped up in a tar->scp->untar for performance
+        # if there are lots of files to be copied/moved
+        for src, dst in zip(srcs, dsts):
+            cmd = [scp, '-p', remote + ':' + src, dst]
+            self.local_executor.run(cmd)
+        if delete:
+            for src in remote_srcs:
+                self.delete_remote(src)
+
+    def delete_remote(self, remote, ignore_error=True):
+        try:
+            self.__execute_command_remote(['rm', '-rf', remote])
+        except:
+            if ignore_error:
+                pass
+            else:
+                raise
+
+    def run(self, command, work_dir='.', env=None):
+        return self.__execute_command_remote(command, work_dir, env)
+
+    def __execute_command_remote(self, command, remote_work_dir='', env=None):
+        remote = self.username + '@' + self.host
+        cmd = [self.ssh_command]
+        cmd += ['-oBatchMode=yes']
+        cmd += [remote]
+        env_cmd = []
+        if env:
+            env_cmd += ['env']
+            env_cmd += ['%s=%s' % (k, v) for k, v in env.items()]
+        remote_cmd = ' '.join(env_cmd + command)
+        if remote_work_dir != '.' and remote_work_dir != '':
+          remote_cmd = 'cd ' + remote_work_dir + ' && ' + remote_cmd
+        cmd += [remote_cmd]
+        return self.local_executor.run(cmd)
+
+    def clean_temps(self):
+        for remote_file in self.remote_temp_files:
+            self.delete_remote(remote_file)
+
+        for remote_dir in self.remote_temp_dirs:
+            self.delete_remote(remote_dir)
+
+
+class DebugExecutor(Executor):
+    def __init__(self, label, chain, before=True, after=True):
+        Executor.__init__(self)
+        self.label = label
+        self.chain = chain
+        self.before = before
+        self.after = after
+
+        for n, f in inspect.getmembers(self):
+            if isinstance(f, types.UnboundMethodType):
+                # Skip meta-functions, decorate everything else
+                if not f.func_name.startswith("__"):
+                    setattr(self, n, self.__decorate(f))
+
+    def __decorate(self, func):
+        def wrapper(*func_args, **func_kwargs):
+            # Get the string representation of the function call
+            arg_names = func.func_code.co_varnames[1:func.func_code.co_argcount]
+            args = func_args[:len(arg_names)]
+            defaults = func.func_defaults or ()
+            args = args + defaults[len(defaults) - (func.func_code.co_argcount - len(args)):]
+            params = zip(arg_names, args)
+            args = func_args[len(arg_names):]
+            if args: params.append(('args', args))
+            if func_kwargs: params.append(('kwargs', func_kwargs))
+            arg_str = '(' + ', '.join('%s = %r' % p for p in params) + ' )'
+            call_str = func.func_name + arg_str
+
+            # Perform the call itself, logging before, after, and anything thrown
+            try:
+                self.__log_before(call_str)
+
+                # Forward the actual call to the chain, rather than calling
+                # func itself, which is a base-class method pointer
+                chain_func = getattr(self.chain, func.func_name)
+                res = chain_func(*func_args, **func_kwargs)
+
+                self.__log_after("%s = %r" % (call_str, res))
+                return res
+            except:
+                self.__log_after("%s raised" % (call_str))
+                raise
+
+        return wrapper
+
+    def __log_before(self, message):
+        if self.before:
+            print "BEFORE: %s.%s" % (self.label, message)
+
+    def __log_after(self, message):
+        if self.after:
+            print "AFTER:  %s.%s" % (self.label, message)
+

test/lit.site.cfg.in

 config.enable_monotonic_clock   = "@LIBCXX_ENABLE_MONOTONIC_CLOCK@"
 config.cxx_abi                  = "@LIBCXX_CXX_ABI_LIBNAME@"
 config.use_sanitizer            = "@LLVM_USE_SANITIZER@"
 config.abi_library_path         = "@LIBCXX_CXX_ABI_LIBRARY_PATH@"
 config.configuration_variant    = "@LIBCXX_LIT_VARIANT@"
 config.target_triple            = "@LIBCXX_TARGET_TRIPLE@"
 config.sysroot                  = "@LIBCXX_SYSROOT@"
 config.gcc_toolchain            = "@LIBCXX_GCC_TOOLCHAIN@"
+config.target_executor          = "@LIBCXX_TARGET_EXECUTOR@"
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@LIBCXX_SOURCE_DIR@/test/lit.cfg")