[libcxx] Allow use of ShTest in libc++ tests along with other changes.

[EricWF]

Summary

This patch allows the use of LIT's ShTest format in the libc++ test suite. ShTests have the suffix '.sh.cpp'. It also introduces a series of other changes. These changes are:

• More functionality including parsing test metadata has been moved into LIT.
• LibcxxTestFormat now supports multi-part suffixes.
• the CXXCompiler functionality has been used to shrink the size of LibcxxTestFormat.
• The recursive loading of the site config has been turned into libcxx.test.config.loadSiteConfig so it can be used with libc++abi.
• Temporary files are now created in the build directory of libc++. This follows how it is down in ShTest.
• not.py was added as a utility executable that mirrors the functionality of LLVM's not executable.
• The first ShTest test was added under test/libcxx/double_include.sh.cpp

[EricWF]

Misc cleanups and fixes.

[jroelofs]

test/libcxx/double_include.sh.cpp

96

I think this needs to be guarded out using that libcpp-has-no-threads thing. Same thing for <future>, <mutex>, <shared_mutex>, and <atomic>.

test/libcxx/test/config.py

25

Is this TODO still relevant?

[EricWF]

Allow switching between the LIT internal shell and the external shell for ShTests. See the added documentation for a description on how to do this.

[EricWF]

Address @jroelofs comments.

[jroelofs]

Regarding not.py: what did you try in terms of trying to import it from lit proper? (I need to figure out a solution to the same problem for loading the libcxx bits from libcxxabi's config).

Also, looks good on my end, though I'd like to hear from @danalbert on it. he's better at catching python-isms than I am.

[EricWF]

In D7073#111017, @jroelofs wrote:

Regarding not.py: what did you try in terms of trying to import it from lit proper? (I need to figure out a solution to the same problem for loading the libcxx bits from libcxxabi's config).

Nothing. In particular because I want not.py to work entirely standalone of LIT. The best solution I can think of to import LIT is to add the path to the LIT module to the PYTHONPATH environment variable when running the ShTests. However I'm not sure there is an easy way to detect where LIT is, even inside our lit.cfg file.

[danalbert]

Definitely run pylint and pep8. Mostly good, though I do have some questions and nits.

test/libcxx/compiler.py

0–1

Blank line?

95

Two args have different style: infile vs object_file. Prefer in_file (or even source_file). Same goes for the public API to this.

102

Alignment. Should run pylint and pep8 to catch these things.

test/libcxx/test/config.py

34

No kidding. I thought we were already safe from this before...

179

Parentheses aren't necessary.

181

Why isn't this just a string comparison?

492

No extra spaces inside parens.

494

Style nit: I'd kill the vertical alignment. PEP8 agrees (https://www.python.org/dev/peps/pep-0008/#pet-peeves)

498

I'm not a fan of the vertical alignment here. If you want to keep things a little cleaner than many calls to append,

sub.extend([
    ('foo', foo),
    ('bar', bar),
])

looks cleaner imo.

514

k, v

test/libcxx/test/format.py

60

Glad to see all the code backing this go away. Thanks!

126

This breaks the expectation of what _clean is used for. In the case of building on one machine but running on another, there's no way to determine which files need to be cleaned from which machine now. (I think in my case we're just ignoring any failures from clean failures, so there's no behavior change, but it's still odd). Why isn't object_path cleaning up after itself anymore?

If there is a good reason for object_path not cleaning up after itself, this is fine for now and I'll clean it up later if I think of a good way to do so.

134

Don't you want to check for '// USE_VERIFY' in contents?

This is slower than it used to be. We no longer stop processing after the first non-empty non-comment line, so this could take a long time on large tests.

135

extra_flags = ['-Xclang', '-verify'] if use_verify else [] is fine for something this simple.

138

Is there some sort of os.null that would work on Windows for this?

(not required for this patch, just thinking)

utils/not/not.py

8

What's wrong with // RUN: ! %rest_of_cmd?

15

Why not? We're already importing other things from lit. (Not going to actually review any of the rest of this file since I expect it's just copied directly.)

[EricWF]

Address @danalbert's comments. Changes to come.

test/libcxx/compiler.py

95

Will change.

102

I always find that pylint is too noisy. I'll try pep8.

test/libcxx/test/config.py

179

Will fix.

181

Yep. I just copied it from the clang lit.cfg but I'll change it in ours.

test/libcxx/test/format.py

126

I just cleaned it up using _clean because _clean does exactly what I need. I'll create another function that does the same thing as _clean and call that instead.

134

Don't you want to check for '// USE_VERIFY' in contents?

The old behavior only checked to see if it was in the line. This is definitely slower but .fail.cpp tests should be very small anyway so it shouldn't make too much of a difference.

I would like to move this into parseIntegratedTestScript but this is the best way to currently do it.
I'll add a TODO.

135

Will do.

138

os.devnull should work. Thanks.

utils/not/not.py

8

! is not available in the LIT internal shell. LLVM and clang use llvm/utils/not for this purpose. This is a simplified version that doesn't introduce a dependency on a bunch of LLVM libraries.

15

Because I have no idea how to find the location of the lit module during configuration and there aren't any easy ways to transfer the lit module location to not.py.

If you have any ideas on how to find the location of the lit module I'll try and pass that information to not.py.

[EricWF]

Implement @danalbert's comments.

[EricWF]

Modify not.py so that it takes --lit-site=path/to/llvm/utils/lit as it's first parameter. This allows it to reuse the functionality in lit.util.
The not substitution now expands to path/to/python path/to/libcxx/utils/not//not.py --lit-site=path/to/llvm/utils/lit which is a bit hairy but still do-able.

[EricWF]

Remove unneeded module imports in not.py.

[EricWF]

Fix some bugs and cleanup how CXXCompiler.compileLinkTwoStep handles the object file.

[jroelofs]

It occurs to me that we ought to have an shtest that exercises not.py. How about something like this?

// -*- C++ -*-
//===----------------------------------------------------------------------===//
//
//                     The LLVM Compiler Infrastructure
//
// This file is dual licensed under the MIT and the University of Illinois Open
// Source Licenses. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//

// RUN: %cxx -c %s %flags %compile_flags %link_flags
// RUN: not %run

int main() {
  return 1;
}

[EricWF]

In D7073#111358, @jroelofs wrote:

It occurs to me that we ought to have an shtest that exercises not.py. How about something like this?

Probably a good idea. We should probably have a directory that contains tests for the test suite if that makes sense. Any idea where that should live?

PS. %run expands to %exec %t.exe so you need to put -o %t.exe in the compile command. I'll write up some documentation on the substitutions once this patch gets through.

// -*- C++ -*-
//===----------------------------------------------------------------------===//
//
//                     The LLVM Compiler Infrastructure
//
// This file is dual licensed under the MIT and the University of Illinois Open
// Source Licenses. See LICENSE.TXT for details.
//
//===----------------------------------------------------------------------===//

// RUN: %cxx -c %s %flags %compile_flags %link_flags
// RUN: not %run

int main() {
  return 1;
}

[jroelofs]

Yo dawg, I do think we should have testsuite tests. test/libcxx/selftest perhaps?

[EricWF]

Added a selftest directory with tests that test the test suite (@jroelofs Yo, Dawg). One of these tests exercises not.py.

[jroelofs]

LGTM as long as @danalbert is also happy with it.

[danalbert]

test/libcxx/compiler.py

6–7

pylint really hates [] as a default argument (and apparently for good reason).

Making the defaults be None and using self.flags = flags or [] looks nice.

(I understand that using self.flags = list(flags) actually works around this problem, but I'd like it if we could change the style so it cuts down on noise for those of us that do run linters.)

118

source_files (not a part of this change, but might as well cause some collateral cleanup).

test/libcxx/test/config.py

336

Use os.path.join.

511

This alignment isn't correct. Align with the first element.

535

Unused.

test/libcxx/test/format.py

30

self and litConfig are unused (this could be a function rather than a method). Is this "overriding" something from lit.formats.FileBasedTest?

35

No parens surrounding conditionals.

76

This is "private" to lit.TestRunner. Should this function be given a public name in lit.TestRunner? Is there some other API that should be used?

128

The last three arguments aren't used.

test/libcxx/util.py

6

Unused.

[danalbert]

When we talked about this you had mentioned that this would make it easier to access the tests that were built for debugging. I don't see anything here that does that. Is that going to be a follow up patch?

test/libcxx/util.py

35

This is really a NullContextManager (nothing is specific to a file name). I'm not sure what a good name for the function would be. Possibly just nullContext?

utils/not/not.py

4

The block should be the module docstring.

13

sys.argv should be treated as constant, imo. Make a copy if you're going to use it this way.

21

Why does LIT use its own home grown version of this? https://docs.python.org/2/distutils/apiref.html#module-distutils.spawn

There's even a comment in the code that says "Would be nice if Python had a lib function for this". Was distutils.spawn.find_executable not available when that was written?

If you use the distutils API (and just use subprocess instead of lit.util.executeCommand for this you can get rid of the need for --lit-site, which makes up most of the gross in this file.

26

This doesn't directly mirror the one in llvm/utils. The other one only considers negative values success if passed --crash.

Aside from that, it's kind of a weird flag imo. I would have made it a separate expect_crash command. not --crash reads to me as "doesn't crash".

[EricWF]

Address first half of @danalbert's comments.

[EricWF]

Address the rest of @danalbert's comments. Mostly surrounding not.py.

[danalbert]

With the docstring nit fixed, LGTM.

test/libcxx/compiler.py

41

Ah, lots of these. Thanks.

utils/not/not.py

6

Module docstrings go at the very top, before imports.

Whole file looks much better now, btw. Thanks. I'd still like it if this wasn't necessary, so we should maybe look at adding ! to the builtin shell at some point, but this is plenty fine for now.

[EricWF]

Answer @danalbert's questions. Thanks for the review.

test/libcxx/compiler.py

6–7

Huh, I had no idea default arguments worked like that in python.

I still think you should copy the list in so you would need to write self.flags = list(flags or []). Which is more complicated that just self.flags = list(flags). I'll make the change to reduce noise but I'm not convinced it is a good one.

test/libcxx/test/config.py

34

Two reasons for this change.

1. In order to turn this into a reusable function that function should probably return. Recursive reloading would prevent this.
2. I actually have plans to introduce a second benchmarking test suite that relies on the same site config file but it can't load the current lit.cfg

test/libcxx/test/format.py

30

It's replacing the method by the same name in lit.formats.FileBasedTest with a small tweak that allows multipart suffixes. I didn't want to make the change in LIT proper because (as previously discussed) we might try and move away from metadata in test names.

76

Maybe? I split _runShTest() from lit.TestRunner.executeShTest() specifically for use with libc++. I'm hesitant to make the function public if we are the only people that need it. There is no other API in lit.TestRunner that would serve our needs.

utils/not/not.py

6

I've looked at the internal shell a bit and it seems that it could use a lot of love. I'll look into adding ! but it is super low on my list.

[EricWF]

Move docstring in not.py and merge with upstream libc++.

test/libcxx/compiler.py

-
 import lit.util
+import libcxx.util
 
 
 class CXXCompiler(object):
-    def __init__(self, path, flags=[], compile_flags=[], link_flags=[], use_ccache=False):
+    def __init__(self, path, flags=[], compile_flags=[], link_flags=[],
+                 use_ccache=False):
         self.path = path
         self.flags = list(flags)
         self.compile_flags = list(compile_flags)

         out, err, rc = lit.util.executeCommand(cmd, env=env, cwd=cwd)
         return cmd, out, err, rc
 
+    def _compileLinkTwoSteps(self, source_file, object_file, out=None,
+                             flags=[], env=None, cwd=None):
+        cmd, output, err, rc = self.compile(source_file, object_file,
+                                            flags=flags, env=env, cwd=cwd)
+        if rc != 0:
+            return cmd, output, err, rc
+        return self.link(object_file, out=out, flags=flags, env=env,
+                         cwd=cwd)
+
+    def compileLinkTwoSteps(self, source_file, out=None, object_file=None,
+                            flags=[], env=None, cwd=None):
+        if not isinstance(source_file, str):
+            raise TypeError('This function only accepts a single input file')
+        # Create, use and delete a temporary object file if none is given.
+        if object_file is None:
+            with libcxx.util.guardedTemporaryFile(suffix='.o') as object_file:
+                return self._compileLinkTwoSteps(source_file, object_file, out,
+                                                 flags, env, cwd)
+        # Othewise compile using the given object file.
+        else:
+            return self._compileLinkTwoSteps(infile, object_file, out, flags,
+                                             env, cwd)
+
     def dumpMacros(self, infiles=None, flags=[], env=None, cwd=None):
         if infiles is None:
             infiles = '/dev/null'

test/libcxx/double_include.sh.cpp

+// -*- C++ -*-
+//===----------------------------------------------------------------------===//
+//
+//                     The LLVM Compiler Infrastructure
+//
+// This file is dual licensed under the MIT and the University of Illinois Open
+// Source Licenses. See LICENSE.TXT for details.
+//
+//===----------------------------------------------------------------------===//
+
+// Test that we can include each header in two TU's and link them together.
+
+// RUN: %cxx -c %s -o %t.first.o %flags %compile_flags
+// RUN: %cxx -c %s -o %t.second.o -DWITH_MAIN %flags %compile_flags
+// RUN: %cxx -o %t.exe %t.first.o %t.second.o %flags %link_flags
+// RUN: %run
+
+#include <algorithm>
+#include <array>
+#include <bitset>
+#include <cassert>
+#include <ccomplex>
+#include <cctype>
+#include <cerrno>
+#include <cfenv>
+#include <cfloat>
+#include <chrono>
+#include <cinttypes>
+#include <ciso646>
+#include <climits>
+#include <clocale>
+#include <cmath>
+#include <codecvt>
+#include <complex>
+#include <complex.h>
+#include <condition_variable>
+#include <csetjmp>
+#include <csignal>
+#include <cstdarg>
+#include <cstdbool>
+#include <cstddef>
+#include <cstdint>
+#include <cstdio>
+#include <cstdlib>
+#include <cstring>
+#include <ctgmath>
+#include <ctime>
+#include <cwchar>
+#include <cwctype>
+#include <deque>
+#include <exception>
+#include <experimental/dynarray>
+#include <experimental/optional>
+#include <experimental/string_view>
+#include <experimental/type_traits>
+#include <experimental/utility>
+#include <ext/hash_map>
+#include <ext/hash_set>
+#include <forward_list>
+#include <fstream>
+#include <functional>
+#include <initializer_list>
+#include <iomanip>
+#include <ios>
+#include <iosfwd>
+#include <iostream>
+#include <istream>
+#include <iterator>
+#include <limits>
+#include <list>
+#include <locale>
+#include <map>
+#include <memory>
+#include <new>
+#include <numeric>
+#include <ostream>
+#include <queue>
+#include <random>
+#include <ratio>
+#include <regex>
+#include <scoped_allocator>
+#include <set>
+#include <sstream>
+#include <stack>
+#include <stdexcept>
+#include <streambuf>
+#include <string>
+#include <strstream>
+#include <system_error>
+#include <tgmath.h>
+#include <tuple>
+#include <typeindex>
+#include <typeinfo>
+#include <type_traits>
+#include <unordered_map>
+#include <unordered_set>
+#include <utility>
+#include <valarray>
+#include <vector>
+
+#ifndef _LIBCPP_HAS_NO_THREADS
+#include <atomic>
+#include <future>
+#include <mutex>
+#include <shared_mutex>
+#include <thread>
+#endif
+
+#if defined(WITH_MAIN)
+int main() {}
+#endif

test/libcxx/test/config.py

 import locale
 import os
 import platform
+import pkgutil
 import re
 import shlex
 import sys

 from libcxx.compiler import CXXCompiler
 
 
+def loadSiteConfig(lit_config, config, param_name, env_name):
+    # We haven't loaded the site specific configuration (the user is
+    # probably trying to run on a test file directly, and either the site
+    # configuration hasn't been created by the build system, or we are in an
+    # out-of-tree build situation).
+    site_cfg = lit_config.params.get(param_name,
+                                     os.environ.get(env_name))
+    if not site_cfg:
+        lit_config.warning('No site specific configuration file found!'
+                           ' Running the tests in the default configuration.')
+    elif not os.path.isfile(site_cfg):
+        lit_config.fatal(
+            "Specified site configuration file does not exist: '%s'" %
+            site_cfg)
+    else:
+        lit_config.note('using site specific configuration at %s' % site_cfg)
+        ld_fn = lit_config.load_config
+
+        # Null out the load_config function so that lit.site.cfg doesn't
+        # recursively load a config even if it tries.
+        # TODO: This is one hell of a hack. Fix it.
+        def prevent_reload_fn(*args, **kwargs):
+            pass
+        lit_config.load_config = prevent_reload_fn
+        ld_fn(config, site_cfg)
+        lit_config.load_config = ld_fn
+
+
 class Configuration(object):
     # pylint: disable=redefined-outer-name
     def __init__(self, lit_config, config):

         self.use_system_cxx_lib = False
         self.use_clang_verify = False
         self.long_tests = None
+        self.execute_external = False
 
         if platform.system() not in ('Darwin', 'FreeBSD', 'Linux'):
             self.lit_config.fatal("unrecognized system")

         self.configure_cxx_library_root()
         self.configure_use_system_cxx_lib()
         self.configure_use_clang_verify()
+        self.configure_execute_external()
         self.configure_ccache()
         self.configure_env()
         self.configure_compile_flags()
         self.configure_link_flags()
         self.configure_sanitizer()
+        self.configure_substitutions()
         self.configure_features()
 
     def print_config_info(self):
         # Print the final compile and link flags.
         self.lit_config.note('Using compiler: %s' % self.cxx.path)
         self.lit_config.note('Using flags: %s' % self.cxx.flags)
-        self.lit_config.note('Using compile flags: %s' % self.cxx.compile_flags)
+        self.lit_config.note('Using compile flags: %s'
+                             % self.cxx.compile_flags)
         self.lit_config.note('Using link flags: %s' % self.cxx.link_flags)
         # Print as list to prevent "set([...])" from being printed.
         self.lit_config.note('Using available_features: %s' %

         return LibcxxTestFormat(
             self.cxx,
             self.use_clang_verify,
+            self.execute_external,
             exec_env=self.env)
 
     def configure_cxx(self):

             self.config.available_features.add('%s-%s.%s' % (
                 cxx_type, maj_v, min_v))
 
-
     def configure_src_root(self):
         self.libcxx_src_root = self.get_lit_conf(
             'libcxx_src_root', os.path.dirname(self.config.test_source_root))

             self.lit_config.note(
                 "inferred use_clang_verify as: %r" % self.use_clang_verify)
 
+    def configure_execute_external(self):
+        # Choose between lit's internal shell pipeline runner and a real shell.
+        # If LIT_USE_INTERNAL_SHELL is in the environment, we use that as the
+        # default value. Otherwise we default to internal on Windows and
+        # external elsewhere, as bash on Windows is usually very slow.
+        use_lit_shell_default = os.environ.get('LIT_USE_INTERNAL_SHELL')
+        if use_lit_shell_default is not None:
+            use_lit_shell_default = use_lit_shell_default != '0'
+        else:
+            use_lit_shell_default = sys.platform == 'win32'
+        # Check for the command line parameter using the default value if it is
+        # not present.
+        use_lit_shell = self.get_lit_bool('use_lit_shell',
+                                          use_lit_shell_default)
+        self.execute_external = not use_lit_shell
+
     def configure_ccache(self):
         use_ccache = self.get_lit_bool('use_ccache', False)
         if use_ccache:

             self.config.available_features.add('long_tests')
 
     def configure_compile_flags(self):
         no_default_flags = self.get_lit_bool('no_default_flags', False)
         if not no_default_flags:
             self.configure_default_compile_flags()
         # Configure extra flags
         compile_flags_str = self.get_lit_conf('compile_flags', '')
         self.cxx.compile_flags += shlex.split(compile_flags_str)
 
     def configure_default_compile_flags(self):
         # Try and get the std version from the command line. Fall back to

             self.cxx.flags += ['-target', self.config.target_triple]
 
     def configure_compile_flags_header_includes(self):
-        self.cxx.compile_flags += ['-I' + self.libcxx_src_root + '/test/support']
+        self.cxx.compile_flags += ['-I' + self.libcxx_src_root
+                                        + '/test/support']
         libcxx_headers = self.get_lit_conf('libcxx_headers',
                                            self.libcxx_src_root + '/include')
         if not os.path.isdir(libcxx_headers):

     def configure_link_flags(self):
         no_default_flags = self.get_lit_bool('no_default_flags', False)
         if not no_default_flags:
             self.cxx.link_flags += ['-nodefaultlibs']
 
             # Configure library path
             self.configure_link_flags_cxx_library_path()
             self.configure_link_flags_abi_library_path()
 
             # Configure libraries
             self.configure_link_flags_cxx_library()
             self.configure_link_flags_abi_library()
             self.configure_extra_library_flags()
 
         link_flags_str = self.get_lit_conf('link_flags', '')
         self.cxx.link_flags += shlex.split(link_flags_str)

             elif san == 'Memory' or san == 'MemoryWithOrigins':
                 self.cxx.flags += ['-fsanitize=memory']
                 if san == 'MemoryWithOrigins':
-                    self.cxx.compile_flags += ['-fsanitize-memory-track-origins']
+                    self.cxx.compile_flags += [
+                        '-fsanitize-memory-track-origins']
                 if llvm_symbolizer is not None:
                     self.env['MSAN_SYMBOLIZER_PATH'] = llvm_symbolizer
                 self.config.available_features.add('msan')

                 self.lit_config.fatal('unsupported value for '
                                       'use_sanitizer: {0}'.format(san))
 
+    def configure_substitutions(self):
+        sub = self.config.substitutions
+        # Configure compiler substitions
+        sub.append(('%cxx', self.cxx.path))
+        # Configure flags substitutions
+        flags_str = ' '.join(self.cxx.flags)
+        compile_flags_str = ' '.join(self.cxx.compile_flags)
+        link_flags_str = ' '.join(self.cxx.link_flags)
+        all_flags = '%s %s %s' % (flags_str, compile_flags_str, link_flags_str)
+        sub.append(('%flags', flags_str))
+        sub.append(('%compile_flags', compile_flags_str))
+        sub.append(('%link_flags', link_flags_str))
+        sub.append(('%all_flags', all_flags))
+        # Add compile and link shortcuts
+        compile_str = (self.cxx.path + ' -o %t.o %s -c ' + flags_str
+                                                         + compile_flags_str)
+        link_str = (self.cxx.path + ' -o %t.exe %t.o ' + flags_str
+                                                       + link_flags_str)
+        assert type(link_str) is str
+        build_str = self.cxx.path + ' -o %t.exe %s ' + all_flags
+        sub.append(('%compile', compile_str))
+        sub.append(('%link', link_str))
+        sub.append(('%build', build_str))
+        # Configure exec prefix substitutions.
+        exec_env_str = 'env ' if len(self.env) != 0 else ''
+        for k, v in self.env.items():
+            exec_env_str += ' %s=%s' % (k, v)
+        # Configure run env substitution.
+        exec_str = ''
+        if self.lit_config.useValgrind:
+            exec_str = ' '.join(self.lit_config.valgrindArgs) + exec_env_str
+        sub.append(('%exec', exec_str))
+        # Configure run shortcut
+        sub.append(('%run', exec_str + ' %t.exe'))
+        # Configure not program substitions
+        python_exe = sys.executable
+        lit_location = ('--lit-site=%s' %
+                        os.path.dirname(pkgutil.get_loader('lit').filename))
+        not_py = os.path.join(self.libcxx_src_root, 'utils', 'not', 'not.py')
+        not_str = '%s %s %s' % (python_exe, not_py, lit_location)
+        sub.append(('not', python_exe + ' ' + not_py + ' ' + lit_location))
+
     def configure_triple(self):
         # Get or infer the target triple.
         self.config.target_triple = self.get_lit_conf('target_triple')

test/libcxx/test/format.py

 import errno
 import os
-import tempfile
 import time
 
-import lit.formats  # pylint: disable=import-error
+import lit.Test        # pylint: disable=import-error
+import lit.TestRunner  # pylint: disable=import-error
+import lit.util        # pylint: disable=import-error
 
+import libcxx.util
 
-class LibcxxTestFormat(lit.formats.FileBasedTest):
+
+class LibcxxTestFormat(object):
     """
     Custom test format handler for use with the test format use by libc++.
 

       FOO.pass.cpp - Executable test which should compile, run, and exit with
                      code 0.
       FOO.fail.cpp - Negative test case which is expected to fail compilation.
+      FOO.sh.cpp   - A test that uses LIT's ShTest format.
     """
 
-    def __init__(self, cxx, use_verify_for_fail, exec_env):
+    def __init__(self, cxx, use_verify_for_fail, execute_external, exec_env):
         self.cxx = cxx
         self.use_verify_for_fail = use_verify_for_fail
+        self.execute_external = execute_external
         self.exec_env = dict(exec_env)
 
+    # TODO: Move this into lit's FileBasedTest
+    def getTestsInDirectory(self, testSuite, path_in_suite,
+                            litConfig, localConfig):
+        source_path = testSuite.getSourcePath(path_in_suite)
+        for filename in os.listdir(source_path):
+            # Ignore dot files and excluded tests.
+            if (filename.startswith('.') or filename in localConfig.excludes):
+                continue
+
+            filepath = os.path.join(source_path, filename)
+            if not os.path.isdir(filepath):
+                if any([filename.endswith(ext)
+                        for ext in localConfig.suffixes]):
+                    yield lit.Test.Test(testSuite, path_in_suite + (filename,),
+                                        localConfig)
+
     def execute(self, test, lit_config):
         while True:
             try:

                 time.sleep(0.1)
 
     def _execute(self, test, lit_config):
-        # Extract test metadata from the test file.
-        requires = []
-        unsupported = []
-        use_verify = False
-        with open(test.getSourcePath()) as f:
-            for ln in f:
-                if 'XFAIL:' in ln:
-                    items = ln[ln.index('XFAIL:') + 6:].split(',')
-                    test.xfails.extend([s.strip() for s in items])
-                elif 'REQUIRES:' in ln:
-                    items = ln[ln.index('REQUIRES:') + 9:].split(',')
-                    requires.extend([s.strip() for s in items])
-                elif 'UNSUPPORTED:' in ln:
-                    items = ln[ln.index('UNSUPPORTED:') + 12:].split(',')
-                    unsupported.extend([s.strip() for s in items])
-                elif 'USE_VERIFY' in ln and self.use_verify_for_fail:
-                    use_verify = True
-                elif not ln.strip().startswith("//") and ln.strip():
-                    # Stop at the first non-empty line that is not a C++
-                    # comment.
-                    break
-
-        # Check that we have the required features.
-        #
-        # FIXME: For now, this is cribbed from lit.TestRunner, to avoid
-        # introducing a dependency there. What we more ideally would like to do
-        # is lift the "requires" handling to be a core lit framework feature.
-        missing_required_features = [
-            f for f in requires
-            if f not in test.config.available_features
-        ]
-        if missing_required_features:
-            return (lit.Test.UNSUPPORTED,
-                    "Test requires the following features: %s" % (
-                        ', '.join(missing_required_features),))
-
-        unsupported_features = [f for f in unsupported
-                                if f in test.config.available_features]
-        if unsupported_features:
-            return (lit.Test.UNSUPPORTED,
-                    "Test is unsupported with the following features: %s" % (
-                        ', '.join(unsupported_features),))
-
-        # Evaluate the test.
-        return self._evaluate_test(test, use_verify, lit_config)
-
-    def _make_report(self, cmd, out, err, rc):  # pylint: disable=no-self-use
-        report = "Command: %s\n" % cmd
-        report += "Exit Code: %d\n" % rc
-        if out:
-            report += "Standard Output:\n--\n%s--\n" % out
-        if err:
-            report += "Standard Error:\n--\n%s--\n" % err
-        report += '\n'
-        return cmd, report, rc
-
-    def _compile(self, output_path, source_path, use_verify=False):
-        extra_flags = []
-        if use_verify:
-            extra_flags += ['-Xclang', '-verify']
-        return self.cxx.compile(source_path, out=output_path, flags=extra_flags)
+        name = test.path_in_suite[-1]
+        is_sh_test = name.endswith('.sh.cpp')
+        is_pass_test = name.endswith('.pass.cpp')
+        is_fail_test = name.endswith('.fail.cpp')
+
+        res = lit.TestRunner.parseIntegratedTestScript(
+            test, require_script=is_sh_test)
+        # Check if a result for the test was returned. If so return that
+        # result.
+        if isinstance(res, lit.Test.Result):
+            return res
+        if lit_config.noExecute:
+            return lit.Test.Result(lit.Test.PASS)
+        # res is not an instance of lit.test.Result. Expand res into its parts.
+        script, tmpBase, execDir = res
+        # Check that we don't have run lines on tests that don't support them.
+        if not is_sh_test and len(script) != 0:
+            lit_config.fatal('Unsupported RUN line found in test %s' % name)
+
+        # Dispatch the test based on its suffix.
+        if is_sh_test:
+            return lit.TestRunner._runShTest(test, lit_config,
+                                             self.execute_external, script,
+                                             tmpBase, execDir)
+        elif is_fail_test:
+            return self._evaluate_fail_test(test, tmpBase, execDir, lit_config)
+        elif is_pass_test:
+            return self._evaluate_pass_test(test, tmpBase, execDir, lit_config)
+        else:
+            # No other test type is supported
+            assert False
 
-    def _link(self, exec_path, object_path):
-        return self.cxx.link(object_path, out=exec_path)
+    def _clean(self, exec_path):  # pylint: disable=no-self-use
+        libcxx.util.cleanFile(exec_path)
 
-    def _compile_and_link(self, exec_path, source_path):
-        object_file = tempfile.NamedTemporaryFile(suffix=".o", delete=False)
-        object_path = object_file.name
-        object_file.close()
+    def _evaluate_pass_test(self, test, tmpBase, execDir, lit_config):
+        source_path = test.getSourcePath()
+        exec_path = tmpBase + '.exe'
+        object_path = tmpBase + '.o'
+        # Create the output directory if it does not already exist.
+        lit.util.mkdir_p(os.path.dirname(tmpBase))
         try:
-            cmd, out, err, rc = self.cxx.compile(source_path, out=object_path)
+            # Compile the test
+            cmd, out, err, rc = self.cxx.compileLinkTwoSteps(
+                source_path, out=exec_path, object_file=object_path,
+                cwd=execDir)
+            compile_cmd = cmd
+            if rc != 0:
+                report = libcxx.util.makeReport(cmd, out, err, rc)
+                report += "Compilation failed unexpectedly!"
+                return lit.Test.FAIL, report
+            # Run the test
+            cmd = []
+            if self.exec_env:
+                cmd.append('env')
+                cmd.append('%s=%s' % (k, v) for k, v in self.exec_env.items())
+            if lit_config.useValgrind:
+                cmd = lit_config.valgrindArgs + cmd
+            cmd.append(exec_path)
+            out, err, rc = lit.util.executeCommand(
+                cmd, cwd=os.path.dirname(source_path))
             if rc != 0:
-                return cmd, out, err, rc
-            return self.cxx.link(object_path, out=exec_path)
+                report = libcxx.util.makeReport(cmd, out, err, rc)
+                report = "Compiled With: %s\n%s" % (compile_cmd, report)
+                report += "Compiled test failed unexpectedly!"
+                return lit.Test.FAIL, report
+            return lit.Test.PASS, ''
         finally:
-            try:
-                os.remove(object_path)
-            except OSError:
-                pass
+            # Note that cleanup of exec_file happens in `_clean()`. If you
+            # override this, cleanup is your reponsibility.
+            libcxx.util.cleanFile(object_path)
+            self._clean(exec_path)
 
-    def _build(self, exec_path, source_path, compile_only=False,
-               use_verify=False):
-        if compile_only:
-            cmd, out, err, rc = self._compile(exec_path, source_path,
-                                              use_verify)
-        else:
-            assert not use_verify
-            cmd, out, err, rc = self._compile_and_link(exec_path, source_path)
-        return self._make_report(cmd, out, err, rc)
-
-    def _clean(self, exec_path):  # pylint: disable=no-self-use
-        try:
-            os.remove(exec_path)
-        except OSError:
-            pass
-
-    def _run(self, exec_path, lit_config, in_dir=None):
-        cmd = []
-        if self.exec_env:
-            cmd.append('env')
-            cmd.extend('%s=%s' % (name, value)
-                       for name, value in self.exec_env.items())
-        cmd.append(exec_path)
-        if lit_config.useValgrind:
-            cmd = lit_config.valgrindArgs + cmd
-        out, err, rc = lit.util.executeCommand(cmd, cwd=in_dir)
-        return self._make_report(cmd, out, err, rc)
-
-    def _evaluate_test(self, test, use_verify, lit_config):
-        name = test.path_in_suite[-1]
+    def _evaluate_fail_test(self, test, tmpBase, execDir, lit_config):
         source_path = test.getSourcePath()
-        source_dir = os.path.dirname(source_path)
-
-        # Check what kind of test this is.
-        assert name.endswith('.pass.cpp') or name.endswith('.fail.cpp')
-        expected_compile_fail = name.endswith('.fail.cpp')
-
-        # If this is a compile (failure) test, build it and check for failure.
-        if expected_compile_fail:
-            cmd, report, rc = self._build('/dev/null', source_path,
-                                          compile_only=True,
-                                          use_verify=use_verify)
-            expected_rc = 0 if use_verify else 1
-            if rc == expected_rc:
-                return lit.Test.PASS, ""
-            else:
-                return (lit.Test.FAIL,
-                        report + 'Expected compilation to fail!\n')
+        # TODO: Move the checking of USE_VERIFY into
+        # lit.TestRunner.parseIntegratedTestScript by adding support for custom
+        # tags.
+        with open(source_path, 'r') as f:
+            contents = f.read()
+        use_verify = 'USE_VERIFY' in contents and self.use_verify_for_fail
+        extra_flags = ['-Xclang', '-verify'] if use_verify else []
+        cmd, out, err, rc = self.cxx.compile(source_path, out=os.devnull,
+                                             flags=extra_flags)
+        expected_rc = 0 if use_verify else 1
+        if rc == expected_rc:
+            return lit.Test.PASS, ''
         else:
-            exec_file = tempfile.NamedTemporaryFile(suffix="exe", delete=False)
-            exec_path = exec_file.name
-            exec_file.close()
-
-            try:
-                cmd, report, rc = self._build(exec_path, source_path)
-                compile_cmd = cmd
-                if rc != 0:
-                    report += "Compilation failed unexpectedly!"
-                    return lit.Test.FAIL, report
-
-                cmd, report, rc = self._run(exec_path, lit_config,
-                                            source_dir)
-                if rc != 0:
-                    report = "Compiled With: %s\n%s" % (compile_cmd, report)
-                    report += "Compiled test failed unexpectedly!"
-                    return lit.Test.FAIL, report
-            finally:
-                # Note that cleanup of exec_file happens in `_clean()`. If you
-                # override this, cleanup is your reponsibility.
-                self._clean(exec_path)
-        return lit.Test.PASS, ""
+            report = libcxx.util.makeReport(cmd, out, err, rc)
+            return (lit.Test.FAIL,
+                    report + 'Expected compilation to fail!\n')

test/libcxx/util.py

+from contextlib import contextmanager
+import os
+import tempfile
+
+import lit.util  # pylint: disable=import-error
+
+
+def cleanFile(filename):
+    try:
+        os.remove(filename)
+    except OSError:
+        pass
+
+
+@contextmanager
+def guardedTemporaryFile(suffix='', prefix='', dir=None):
+    handle, name = tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=dir)
+    os.close(handle)
+    yield name
+    try:
+        os.remove(name)
+    except OSError:
+        pass
+
+
+def makeReport(cmd, out, err, rc):
+    report = "Command: %s\n" % cmd
+    report += "Exit Code: %d\n" % rc
+    if out:
+        report += "Standard Output:\n--\n%s--\n" % out
+    if err:
+        report += "Standard Error:\n--\n%s--\n" % err
+    report += '\n'
+    return report

test/lit.cfg

 config.name = 'libc++'
 
 # suffixes: A list of file extensions to treat as test files.
-config.suffixes = ['.cpp']
+config.suffixes = ['.pass.cpp', '.fail.cpp', '.sh.cpp']
 
 # test_source_root: The root path where tests are located.
 config.test_source_root = os.path.dirname(__file__)
 
 # Infer the test_exec_root from the libcxx_object root.
-libcxx_obj_root = getattr(config, 'libcxx_obj_root', None)
-if libcxx_obj_root is not None:
-    config.test_exec_root = os.path.join(libcxx_obj_root, 'test')
+obj_root = getattr(config, 'libcxx_obj_root', None)
 
 # Check that the test exec root is known.
-if config.test_exec_root is None:
-    # Otherwise, we haven't loaded the site specific configuration (the user is
-    # probably trying to run on a test file directly, and either the site
-    # configuration hasn't been created by the build system, or we are in an
-    # out-of-tree build situation).
-    site_cfg = lit_config.params.get('libcxx_site_config',
-                                     os.environ.get('LIBCXX_SITE_CONFIG'))
-    if not site_cfg:
-        lit_config.warning('No site specific configuration file found!'
-                           ' Running the tests in the default configuration.')
-        # TODO: Set test_exec_root to a temporary directory where output files
-        # can be placed. This is needed for ShTest.
-    elif not os.path.isfile(site_cfg):
-        lit_config.fatal(
-            "Specified site configuration file does not exist: '%s'" %
-            site_cfg)
-    else:
-        lit_config.note('using site specific configuration at %s' % site_cfg)
-        lit_config.load_config(config, site_cfg)
-        raise SystemExit()
+if obj_root is None:
+    import libcxx.test.config
+    libcxx.test.config.loadSiteConfig(lit_config, config, 'libcxx_site_config',
+                                      'LIBCXX_SITE_CONFIG')
+    obj_root = getattr(config, 'libcxx_obj_root', None)
+    if obj_root is None:
+        import tempfile
+        obj_root = tempfile.mkdtemp(prefix='libcxx-testsuite-')
+        lit_config.warning('Creating temporary directory for object root: %s' %
+                           obj_root)
+
+config.test_exec_root = os.path.join(obj_root, 'test')
 
 cfg_variant = getattr(config, 'configuration_variant', 'libcxx')
 if cfg_variant:

utils/not/not.py

+import sys
+
+# not.py is a utility for inverting the return code of commands.
+# It acts similar to llvm/utils/not.
+# ex: python /path/to/not.py --lit-path=path/to/lit' echo hello
+#     echo $? // (prints 1)
+
+
+def main():
+    import lit.util
+    argv = sys.argv
+    del argv[0]
+    if len(argv) > 0 and argv[0] == '--crash':
+        del argv[0]
+        expectCrash = True
+    else:
+        expectCrash = False
+    if len(argv) == 0:
+        return 1
+    prog = lit.util.which(argv[0])
+    if prog is None:
+        sys.stderr.write('Failed to find program %s' % argv[0])
+        return 1
+    out, err, rc = lit.util.executeCommand(argv)
+    if rc == 0 and not expectCrash:
+        return 1
+    else:
+        return 0
+
+
+if __name__ == '__main__':
+    import site
+    assert len(sys.argv) >= 2
+    lit_site = sys.argv[1]
+    del sys.argv[1]
+    assert lit_site.startswith('--lit-site=')
+    lit_site = lit_site[len('--lit-site='):]
+    site.addsitedir(lit_site)
+    exit(main())

www/lit_usage.html

 </p>
 
 <p>
+<h3 class="lit-option">use_lit_shell=&lt;bool&gt;</h3>
+<blockquote class="lit-option-desc">
+Enable or disable the use of LIT's internal shell in ShTests. If the enviroment
+variable <code>LIT_USE_INTERNAL_SHELL</code> is present then that is used as the
+default value. Otherwise the default value is <code>True</code> on Windows and
+<code>False</code> on every other platform.
+</blockquote>
+</p>
+
+<p>
 <h3 class="lit-option">no_default_flags=&lt;bool&gt;</h3>
 <blockquote class="lit-option-desc">
 <b>Default: </b><code>False</code></br>