[libcxx] Add numerous options to libc++ LIT test suite configuration.

[EricWF]

Summary

In order to fully replace the testit script we need to update LIT so it provides the same functionality.
This patch adds a number of different configuration options to LIT to do that. It also adds documentation for all of the command line parameters that LIT supports.

Generic options added:

• libcxx_headers
• libcxx_library
• compile_flags

Generic options modified:

• link_flags: Changed from overriding the default args to adding extra args instead (to match compile flags)
• use_sanitizer: Renamed from llvm_use_sanitizer

Please see the added documentation for more information about the switches. As for the actual documentation I'm not sure if it should be kept in libc++ forever since it adds an undue maintenance burden, but I think it should be added for the time being while the changes are new. I'm verify unskilled with HTML so if the documentation needs any changes please let me know.

Hopefully this will kill testit.

[alexfh]

test/lit.cfg

578

I wouldn't like this to depend on the default set of checks. And I don't think you want all static analyzer checks. I'd better change this to:

-checks=-*,llvm*,clang-analyzer*,-clang-analyzer-alpha*

Also, could you just use an appropriate .clang-tidy file instead?

[EricWF]

Thanks for the information on the default clang tidy settings, I wasn't quite sure what to use :D

The reason I didn't use a clang-tidy is because much of the clang-tidy invocation is sensative to information that could change from run to run due to different configurations.
I'll take a look into it further to see if there is any information that is consistent.

[EricWF]

Update patch so it applies cleanly to TOT. Also change the default clang-tidy checkers. I didn't add a .clang-tidy file since a user may want to disable the default checkers by overriding them on the command line.

[alexfh]

test/lit.cfg

572

It should be -checks=-*,llvm*,... i.e. the first item in the list should be "-*" which means "disable all checks".

[EricWF]

Woops. Fix clang-tidy default args again. Thanks.

[EricWF]

Update docs for last change. Sorry for the spam.

[alexfh]

I know nothing about libcxx testing, but clang-tidy-related bits look good now ;) (assuming you have run this already and it works).

[EricWF]

Taking this offline.

Thanks for the input. Using clang-tidy is a nice feature. (even though
every single test fails w/ clang-tidy and the default options).
I guess we have plenty of work to do.

/Eric

[EricWF]

Add support for building and generating code coverage using lcov and genhtml. To produce code coverage you must:

• configure with '-DLIBCXX_GENERATE_COVERAGE=ON'
• make
• make check-libcxx
• make check-libcxx-coverage

The resulting HTML can be found in <build-libcxx>/test/coverage/libcxx_coverage

This also adds support for UNSUPPORTED-XFAIL and REQUIRES-XFAIL tags that ensure compilation fail but report the test as UNSUPPORTED instead of XFAIL.

[danalbert]

Mega patches make me sad. I see 5 distinct changes here:

1. Add missing testit features.
2. scan-build support.
3. clang-tidy support.
4. Coverage support.
5. REQUIRES-XFAIL/UNSUPPORTED-XFAIL.

Please split up this change into separate patches. Voltron-esque patches are hard to review, since we have to context switch so many times, and they're less useful when looking at git history because it isn't always immediately obvious which of the many things a given change was for.

cmake/Modules/LibcxxCodeCoverage.cmake

2 ↗

(On Diff #15968)

This looks like a leftover.

10 ↗

(On Diff #15968)

iirc --coverage implies -fprofile-arcs and -ftest-coverage.

13 ↗

(On Diff #15968)

Can move these two checks up to where find_program is called

20 ↗

(On Diff #15968)

MAKE_DIRECTORY?

22 ↗

(On Diff #15968)

Don't we already have a cxx target?

37 ↗

(On Diff #15968)

Why?

test/lit.cfg

33

Python style is kwarg=value (no space)

47

What was it before if not a string?

49

I don't think this does what you think it does.

You're probably looking for coverage_flags.split(' ')

50

Same as 47

51

Is this actually interesting to run on the tests? I would think this is the kind of thing we should be running on libc++ itself.

53

Same as 49

54

Same as 51

55

Same as 49

57

Diff fail :(

I think the new name is actually a better match for the old behavior (since this doesn't actually do the reporting now). I'd change the name back.

Hard to see what else changed... (should be more clear if the name changes back).

105

Do we actually need these? It looks like it's just complicating things.

183

Variable style changed here.

We're kind of inconsistent here anyway, but we should probably drift toward actually matching PEP8.

200

Hmm. I thought the driver would dump the gcno in the same directory as the output file (ditto for the gcd). What is this for?

234

spaces after ,

306

Why do we need on/off? Doesn't that just mean cmake is missing a pythonize_bool?

313

Why do we have two of these?

496

Should say which one is being used if we aren't making this an error

501

Doesn't rpath need an argument?

572

Don't think you meant to unalign them.

597

I don't quite understand why '' means yes, but 'true' means no. Shouldn't this just be

if not self.use_scan_build:
    ...
else:
    ...

? It changes the meaning slightly, but I think this behavior makes more sense. Am I missing something?

[EricWF]

In D5877#18, @danalbert wrote:

Mega patches make me sad. I see 5 distinct changes here:

1. Add missing testit features.
2. scan-build support.
3. clang-tidy support.
4. Coverage support.
5. REQUIRES-XFAIL/UNSUPPORTED-XFAIL.

Please split up this change into separate patches. Voltron-esque patches are hard to review, since we have to context switch so many times, and they're less useful when looking at git history because it isn't always immediately obvious which of the many things a given change was for.

Sorry about that. I'll split them up. I was hoping to get away with it this once. It's hard to maintain and merge 5 separate patches that exist in such a small space.

cmake/Modules/LibcxxCodeCoverage.cmake

2 ↗

(On Diff #15968)

lcov depends on gcov. I thought we should check if gcov is found and issue a hard error otherwise.

10 ↗

(On Diff #15968)

Yep. My mistake. Thanks.

20 ↗

(On Diff #15968)

It creates a directory where all of the coverage files will live.

22 ↗

(On Diff #15968)

Yep. This adds a command that executes before the cxx target.

37 ↗

(On Diff #15968)

In case the resulting HTML is hosted on a web server this minimizes the permissions.

test/lit.cfg

47

I was being dumb.

49

coverage_flags should already be a list. This performs a copy of the list so we have our own.

51

Yes. The tests instantiate the templates. Without the instantiations the static analyzer has nothing to do.

57

Ok. Will do.

105

I'll pull these out for now, but the change is something I would like. It gives the ability to file the test as unsupported while ensuring it fails. It would be helpful for ignoring the std::tuple tests in c++03 mode.

200

The driver dumps the gcno file in the cwd used to invoke the compiler. Since the gcno will be named <test-name>.gcno and not the unique executable output name we need to mirror the test directory structure to ensure that gcno files for each test don't conflict.

306

This input could also come from the command line. I'm not opposed to only using one name for true and false but this gives flexibility.

313

Different semantics. I'm not a fan of the get_lit_bool semantics.

get_lit_bool:

• None -> None
• "" -> False

get_lit_switch:

• None -> False
• "" -> True

get_lit_switch allows the no-argument case to return True, and always returns a bool.

496

I'll make it an error.

501

It gets its argument from the second -Wl flag but I'll join them for clarity.

597

The semantics of use_scan_build are intended to be:

--param=use_scan_build[=<path/to/scan-build>]

Where the argument defaults to scan-build if omitted.

[EricWF]

I've stripped the patch down to a minimal set of changes so that it has the same functionality as the testit script.

I've removed:

• UNSUPPORTED-XFAIL and REQUIRES-XFAIL
• support for scan-build
• support for clang-tidy
• support for code coverage.

[EricWF]

Changed enable_warnings so it sets -Werror. Otherwise the switch is pretty useless.

[danalbert]

A bunch of style nits and a few doc changes, but otherwise looks good.

test/lit.cfg

41

Spaces after commas.

42

Why the docstring? I see you didn't actually put them here, but what's the point?

Same goes for the rest of them.

55

Spaces after commas.

128

Spaces after commas.

142

Spaces after commas.

154

Again.

219

Don't need the trailing \ when it's inside parentheses.

344

Alignment.

359

Agreed. Fatal makes more sense.

370

Alignment.

www/lit_usage.html

16

Space after colon.

54

s/it's/its/

62

s/"/'/

136

Should say how the list is delimited. Same for the others.

143

1z should probably come after 14.

[mclow.lists]

General comments:

1. I'm liking how this is going. A lot.
2. The symlink thing in using lit seems to be misguided to me. You should be able to specify a lit.site.cfg file on the command line. This will make it much easier to support multiple test configurations. I can think of 32 right off the bat: c++03/11/14/1z vs. 32/64 bit vs. No Sanitizer/ASAN/TSAN/UBSAN

3. Did I mention that I like the fast that it's much faster than testit? :-)

[EricWF]

Fix more of @danalbert's comments. I would like to go forward with this patch even though there is a conflicting patch out against LIT.

[danalbert]

LGTM with the one doc change.

www/lit_usage.html

128

I had meant explicitly state that it is space delimited.

[EricWF]

I've rewritten the patch against TOT. There are a couple of changes.

• Removed ability to enable warnings for now.
• Added documentation of the libcxx_site_config option.
• Added documentation about how compile_flags and link_flags are delimited.

[danalbert]

LGTM. Thanks!

test/lit.cfg

     """
 
     def __init__(self, cxx_under_test, use_verify_for_fail,
-                 cpp_flags, ld_flags, exec_env):
+                 cpp_flags, ld_flags, exec_env,
+                 use_scan_build = None,
+                 scan_build_output = None,
+                 scan_build_args = [],
+                 use_clang_tidy = None,
+                 clang_tidy_args = []):
         self.cxx_under_test = cxx_under_test
         self.use_verify_for_fail = use_verify_for_fail
         self.cpp_flags = list(cpp_flags)
         self.ld_flags = list(ld_flags)
         self.exec_env = dict(exec_env)
+        self.use_scan_build = use_scan_build
+        self.scan_build_output = scan_build_output
+        self.scan_build_args = list(scan_build_args)
+        self.use_clang_tidy = use_clang_tidy
+        self.clang_tidy_args = list(clang_tidy_args)
 
     def execute_command(self, command, in_dir=None):
         kwargs = {

             exec_file.close()
 
             try:
+                if self.use_clang_tidy:
+                    status,report = self._execute_clang_tidy_test(source_path)
+                    if status != lit.Test.PASS:
+                        return status,report
+
                 compile_cmd = [self.cxx_under_test, '-o', exec_path,
                        source_path] + self.cpp_flags + self.ld_flags
-                cmd = compile_cmd
+                if self.use_scan_build:
+                    cmd = [self.use_scan_build, '--status-bugs',
+                           '-o', self.scan_build_output] + self.scan_build_args
+                else:
+                    cmd = []
+                cmd += compile_cmd
                 out, err, exitCode = self.execute_command(cmd)
-                if exitCode != 0:
+                if exitCode != 0 or (self.use_scan_build and
+                                    (not os.path.exists(exec_path)
+                                     or os.path.getsize(exec_path) == 0)):
                     report = """Command: %s\n""" % ' '.join(["'%s'" % a
                                                              for a in cmd])
                     report += """Exit Code: %d\n""" % exitCode

                     pass
         return lit.Test.PASS, ""
 
+    def _execute_clang_tidy_test(self, source_file):
+        cmd = [self.use_clang_tidy] + self.clang_tidy_args + [source_file]
+        cmd += ['--'] + self.cpp_flags
+        out, err, exitCode = self.execute_command(cmd)
+        if exitCode != 0 or out.strip():
+            report = """Command: %s\n""" % ' '.join(["'%s'" % a
+                                                     for a in cmd])
+            report += """Exit Code: %d\n""" % exitCode
+            if out:
+                report += """Standard Output:\n--\n%s--""" % out
+            if err:
+                report += """Standard Error:\n--\n%s--""" % err
+            report += "\n\nCompilation failed unexpectedly!"
+            return lit.Test.FAIL, report
+        return lit.Test.PASS, ""
+
 
 class Configuration(object):
     def __init__(self, lit_config, config):

         self.link_flags = []
         self.use_system_lib = False
         self.use_clang_verify = False
+        self.use_scan_build = None
+        self.scan_build_output = None
+        self.scan_build_args = []
+        self.use_clang_tidy = None
+        self.clang_tidy_args = []
 
         if platform.system() not in ('Darwin', 'FreeBSD', 'Linux'):
             self.lit_config.fatal("unrecognized system")

         conf = self.get_lit_conf(name)
         if conf is None:
             return None
-        if conf.lower() in ('1', 'true'):
+        if conf.lower() in ('1', 'true', 'on'):
+            return True
+        if conf.lower() in ('', '0', 'false', 'off'):
+            return False
+        self.lit_config.fatal(
+            "parameter '{}' should be true or false".format(name))
+
+    def get_lit_switch(self, name):
+        s = self.get_lit_conf(name)
+        if s is None:
+            return False
+        elif s.lower() in ('', '1', 'true', 'on'):
             return True
-        if conf.lower() in ('', '0', 'false'):
+        elif s.lower() in ('0', 'false', 'off'):
             return False
         self.lit_config.fatal(
             "parameter '{}' should be true or false".format(name))

         self.configure_obj_root()
         self.configure_use_system_lib()
         self.configure_use_clang_verify()
-        self.configure_env()
         self.configure_std_flag()
         self.configure_compile_flags()
         self.configure_link_flags()
         self.configure_sanitizer()
+        self.configure_warnings()
+        self.configure_scan_build()
+        self.configure_clang_tidy()
         self.configure_features()
 
     def get_test_format(self):

             self.use_clang_verify,
             cpp_flags=['-nostdinc++'] + self.compile_flags,
             ld_flags=['-nodefaultlibs'] + self.link_flags,
-            exec_env=self.env)
+            exec_env=self.env,
+            use_scan_build=self.use_scan_build,
+            scan_build_output=self.scan_build_output,
+            scan_build_args=self.scan_build_args,
+            use_clang_tidy=self.use_clang_tidy,
+            clang_tidy_args=self.clang_tidy_args)
 
     def configure_cxx(self):
         # Gather various compiler parameters.

 
     def configure_use_clang_verify(self):
         '''If set, run clang with -verify on failing tests.'''
-        self.use_clang_verify = self.get_lit_bool('use_clang_verify')
-        if self.use_clang_verify is None:
-            # TODO: Default this to True when using clang.
-            self.use_clang_verify = False
-            self.lit_config.note(
-                "inferred use_clang_verify as: %r" % self.use_clang_verify)
+        self.use_clang_verify = self.get_lit_switch('use_clang_verify')
+
 
     def configure_features(self):
         additional_features = self.get_lit_conf('additional_features')

             self.compile_flags += ['-D_LIBCPP_HAS_NO_MONOTONIC_CLOCK']
 
     def configure_compile_flags(self):
+        # Check if libcxx_headers is defined. If it is then use that as
+        # the location for the libcxx headers.
+        libcxx_headers = self.get_lit_conf('libcxx_headers',
+                                           self.src_root + '/include')
         # Configure extra compiler flags.
-        self.compile_flags += ['-I' + self.src_root + '/include',
+        self.compile_flags += ['-I' + libcxx_headers,
                                '-I' + self.src_root + '/test/support']
         if sys.platform == 'linux2':
             self.compile_flags += ['-D__STDC_FORMAT_MACROS',
                                    '-D__STDC_LIMIT_MACROS',
                                    '-D__STDC_CONSTANT_MACROS']
+        # Add extra compile flags as specified on the command line.
+        extra_flags = self.get_lit_conf('compile_flags')
+        if extra_flags:
+            self.compile_flags += shlex.split(extra_flags)
 
     def configure_link_flags(self):
-        # Configure library search paths
-        abi_library_path = self.get_lit_conf('abi_library_path', '')
-        self.link_flags += ['-L' + self.obj_root + '/lib']
-        if not self.use_system_lib:
-            self.link_flags += ['-Wl,-rpath', '-Wl,' + self.obj_root + '/lib']
+        # Configure libc++ library
+        libcxx_library = self.get_lit_conf('libcxx_library')
+        # Ignore and warn if use_system_lib is true and libcxx_library is specified.
+        if self.use_system_lib and libcxx_library:
+            self.lit_config.warning(
+                'Conflicting options detected: libcxx_library and use_system_lib')
+            libcxx_library = None
+        # Link to the specific libc++ library if specified. Otherwise link to -lc++
+        # and add the build directory as a search path if use_system_lib is False.
+        if libcxx_library:
+            self.link_flags += [libcxx_library, '-Wl,-rpath',
+                                '-Wl,' + os.path.dirname(libcxx_library)]
+        else:
+            self.link_flags += ['-lc++']
+            if not self.use_system_lib:
+                self.link_flags += ['-L' + self.obj_root + '/lib', '-Wl,-rpath',
+                                    '-Wl,' + self.obj_root + '/lib']
+
+        # Add a search path for the ABI library path if specified.
+        abi_library_path = self.get_lit_conf('abi_library_path')
         if abi_library_path:
             self.link_flags += ['-L' + abi_library_path,
                                 '-Wl,-rpath', '-Wl,' + abi_library_path]
-        # Configure libraries
-        self.link_flags += ['-lc++']
-        link_flags_str = self.get_lit_conf('link_flags')
-        if link_flags_str is None:
-            cxx_abi = self.get_lit_conf('cxx_abi', 'libcxxabi')
-            if cxx_abi == 'libstdc++':
-                self.link_flags += ['-lstdc++']
-            elif cxx_abi == 'libsupc++':
-                self.link_flags += ['-lsupc++']
-            elif cxx_abi == 'libcxxabi':
-                self.link_flags += ['-lc++abi']
-            elif cxx_abi == 'libcxxrt':
-                self.link_flags += ['-lcxxrt']
-            elif cxx_abi == 'none':
-                pass
-            else:
-                self.lit_config.fatal(
-                    'C++ ABI setting %s unsupported for tests' % cxx_abi)
-
-            if sys.platform == 'darwin':
-                self.link_flags += ['-lSystem']
-            elif sys.platform == 'linux2':
-                self.link_flags += ['-lgcc_eh', '-lc', '-lm', '-lpthread',
-                                    '-lrt', '-lgcc_s']
-            elif sys.platform.startswith('freebsd'):
-                self.link_flags += ['-lc', '-lm', '-pthread', '-lgcc_s']
-            else:
-                self.lit_config.fatal("unrecognized system: %r" % sys.platform)
 
-            self.lit_config.note(
-                "inferred link_flags as: %r" % self.link_flags)
+        cxx_abi = self.get_lit_conf('cxx_abi', 'libcxxabi')
+        if cxx_abi == 'libstdc++':
+            self.link_flags += ['-lstdc++']
+        elif cxx_abi == 'libsupc++':
+            self.link_flags += ['-lsupc++']
+        elif cxx_abi == 'libcxxabi':
+            self.link_flags += ['-lc++abi']
+        elif cxx_abi == 'libcxxrt':
+            self.link_flags += ['-lcxxrt']
+        elif cxx_abi == 'none':
+            pass
+        else:
+            self.lit_config.fatal(
+                'C++ ABI setting %s unsupported for tests' % cxx_abi)
+
+        if sys.platform == 'darwin':
+            self.link_flags += ['-lSystem']
+        elif sys.platform == 'linux2':
+            self.link_flags += ['-lgcc_eh', '-lc', '-lm', '-lpthread',
+                                '-lrt', '-lgcc_s']
+        elif sys.platform.startswith('freebsd'):
+            self.link_flags += ['-lc', '-lm', '-pthread', '-lgcc_s']
+        else:
+            self.lit_config.fatal("unrecognized system: %r" % sys.platform)
+
+        self.lit_config.note(
+            "inferred link_flags as: %r" % self.link_flags)
+
+        link_flags_str = self.get_lit_conf('link_flags')
         if link_flags_str:
             self.link_flags += shlex.split(link_flags_str)
 

         self.config.available_features.add(std)
 
     def configure_sanitizer(self):
-        san = self.get_lit_conf('llvm_use_sanitizer', '').strip()
-        if san:
+        san = self.get_lit_conf('use_sanitizer').strip()
+        if san and san != 'None':
             self.compile_flags += ['-fno-omit-frame-pointer']
             if not sys.platform.startswith('freebsd'):
                 self.link_flags += ['-ldl']

                 self.config.available_features.add('msan')
             elif san == 'Undefined':
                 self.compile_flags += ['-fsanitize=undefined',
-                                       '-fno-sanitize=vptr,function',
-                                       '-fno-sanitize-recover']
                 self.config.available_features.add('ubsan')
             else:
                 self.lit_config.fatal('unsupported value for '
-                                      'libcxx_use_san: {0}'.format(san))
+                                      'use_sanitizer: {0}'.format(san))
+
+    def configure_warnings(self):
+        enable_warnings = self.get_lit_bool('enable_warnings')
+        if enable_warnings:
+            self.lit_config.note('Enabling warnings')
+            self.compile_flags += ['-Wall', '-Wextra', '-pedantic',
+                                   '-D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER']
+
+    def configure_scan_build(self):
+        self.use_scan_build = self.get_lit_conf('use_scan_build')
+        if self.use_scan_build is None:
+            return
+        if self.use_scan_build == '':
+            self.use_scan_build = 'scan-build'
+        self.lit_config.note(
+            'Using scan-build: %s' % self.use_scan_build)
+        output = self.get_lit_conf('scan_build_output')
+        if output is None:
+            output = tempfile.mkdtemp(suffix='', prefix='libcxx-scan-build-')
+        self.lit_config.note('Writing scan-build output to: %s' % output)
+        self.scan_build_output = output
+        args = self.get_lit_conf('scan_build_args')
+        if args:
+            args = shlex.split(args)
+        else:
+            args = ['-analyze-headers']
+        self.lit_config.note('Using scan-build args: %r' % args)
+        self.scan_build_args = args
+
+    def configure_clang_tidy(self):
+        self.use_clang_tidy = self.get_lit_conf('use_clang_tidy')
+        if self.use_clang_tidy is None:
+            return
+        if self.use_clang_tidy == '':
+            self.use_clang_tidy = 'clang-tidy'
+        self.lit_config.note(
+            'Using clang-tidy: %s' % self.use_clang_tidy)
+        args = self.get_lit_conf('clang_tidy_args')
+        if args is not None:
+            self.clang_tidy_args = shlex.split(args)
+        else:
+            self.clang_tidy_args = ["-checks=-*,llvm*,clang-analyzer*,-clang-analyzer-alpha*",
+                                    "-header-filter='.*'"]
+        self.lit_config.note('Using clang-tidy args: %r' % self.clang_tidy_args)
+
 
     def configure_triple(self):
         # Get or infer the target triple.

             self.lit_config.note(
                 "inferred target_triple as: %r" % self.config.target_triple)
 
-    def configure_env(self):
-        # Configure extra linker parameters.
-        if sys.platform == 'darwin':
-            if not self.use_system_lib:
-                self.env['DYLD_LIBRARY_PATH'] = os.path.join(self.obj_root,
-                                                             'lib')
-
 
 # name: The name of this test suite.
 config.name = 'libc++'

test/lit.site.cfg.in

 config.python_executable     = "@PYTHON_EXECUTABLE@"
 config.enable_shared         = @LIBCXX_ENABLE_SHARED@
 config.cxx_abi               = "@LIBCXX_CXX_ABI_LIBNAME@"
-config.llvm_use_sanitizer    = "@LLVM_USE_SANITIZER@"
 config.abi_library_path      = "@LIBCXX_CXX_ABI_LIBRARY_PATH@"
+config.use_sanitizer         = "@LLVM_USE_SANITIZER@"
 
 # Let the main config do the real work.
 lit_config.load_config(config, "@LIBCXX_SOURCE_DIR@/test/lit.cfg")

www/index.html

 <li><a href="type_traits_design.html"><tt>&lt;type_traits&gt;</tt></a></li>
 <li><a href="http://cplusplusmusings.wordpress.com/2012/07/05/clang-and-standard-libraries-on-mac-os-x/">Excellent notes by Marshall Clow</a></li>
 <li><a href="debug_mode.html">Status of debug mode</a></li>
+<li><a href="lit_usage.html">Using LIT on the command line</a></li>
 </ul>
 
 </div>

www/lit_usage.html

+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
+          "http://www.w3.org/TR/html4/strict.dtd">
+<!-- Material used from: HTML 4.01 specs: http://www.w3.org/TR/html401/ -->
+<html>
+<head>
+  <META http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
+  <title>Testing libc++ using LIT</title>
+  <link type="text/css" rel="stylesheet" href="menu.css">
+  <link type="text/css" rel="stylesheet" href="content.css">
+  <style>
+    .lit-option {
+      padding-top: 0.5em;
+      margin-bottom: 0.0em;
+      font-size: medium;
+      color:#2d58b7
+    }
+    .lit-option-desc {
+      display: block;
+      margin-top: 0em;
+      margin-bottom: 0em;
+      margin-left: 20px;
+      margin-right: 20px;
+    }
+  </style>
+</head>
+
+<body>
+<div id="menu">
+  <div>
+    <a href="http://llvm.org/">LLVM Home</a>
+  </div>
+
+  <div class="submenu">
+    <label>libc++ Info</label>
+    <a href="/index.html">About</a>
+  </div>
+
+  <div class="submenu">
+    <label>Quick Links</label>
+    <a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev">cfe-dev</a>
+    <a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits">cfe-commits</a>
+    <a href="http://llvm.org/bugs/">Bug Reports</a>
+    <a href="http://llvm.org/svn/llvm-project/libcxx/trunk/">Browse SVN</a>
+    <a href="http://llvm.org/viewvc/llvm-project/libcxx/trunk/">Browse ViewVC</a>
+  </div>
+</div>
+
+<div id="content">
+<!--*********************************************************************-->
+                     <h1>Testing libc++ using LIT</h1>
+<!--*********************************************************************-->
+<p>
+libc++ uses LIT to configure and run it's tests. The primary way to run the
+libc++ tests is by using <code>make check-libcxx</code>. However since libc++
+can be used in any number of possible configurations it is important to
+customize the way LIT builds and runs the tests. This guide provides
+information on how to use LIT directly to test libc++.
+</p>
+<p>
+Documentation for LIT can be found
+<a href='http://llvm.org/docs/CommandGuide/lit.html'>here</a>.
+</p>
+
+<!--*********************************************************************-->
+                        <h2>Getting Started</h2>
+<!--*********************************************************************-->
+<p>
+After building libc++ use the following commands before you start using LIT to
+test.
+</p>
+<ul>
+  <li><code>alias lit='python path/to/llvm/utils/lit/lit.py'</code></li>
+  <li><code>ln -s path/to/libcxx-build/test/lit.site.cfg path/to/libcxx/test/lit.site.cfg</code></li>
+</ul>
+<p>
+You can now run the libc++ tests by running:
+</p>
+<ul>
+  <li><code>cd path/to/libcxx</code></li>
+  <li><code>lit -sv ./test</code></li>
+</ul>
+<p>
+To only run a subsection of the tests use:
+<ul>
+  <li><code>lit -sv test/numerics # Run only the numeric tests</code></li>
+</ul>
+
+<!--*********************************************************************-->
+                     <h2>Customization Options</h2>
+<!--*********************************************************************-->
+<p>
+libc++'s testsuite provides multiple options to configure the way the tests
+are build and run. To use these options you pass them on the LIT command line
+as <code>--param NAME</code> or <code>--param NAME=VALUE</code>. Some options
+have default values specified during CMake's configuration. Passing the option
+on the command line will override the default.
+</p>
+
+<p>
+<h3 class="lit-option">libcxx_headers=&lt;path/to/headers&gt;</h3>
+<blockquote class="lit-option-desc">
+Specify the libc++ headers that are tested. By default the headers in the source
+tree are used.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">libcxx_library=&lt;path/to/libc++.so&gt;</h3>
+<blockquote class="lit-option-desc">
+Specify the libc++ library that is tested. By default the library in the build
+directory is used. This option has no effect if <code>use_system_lib</code> is provided.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">use_system_lib=&lt;bool&gt;</h3>
+<blockquote class="lit-option-desc">
+<b>Default: </b><code>False</code></br>
+Enable or disable testing against the installed version of libc++ library.
+Note: This does not use the installed headers.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">compile_flags="&lt;list-of-args&gt;"</h3>
+<blockquote class="lit-option-desc">
+Specify an additional list of compile flags to use.
+Note: This options should not be used to change the standard version used.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">link_flags="&lt;list-of-args&gt;"</h3>
+<blockquote class="lit-option-desc">
+Specify an additional list of linker flags to use.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">std=&lt;standard version&gt;</h3>
+<blockquote class="lit-option-desc">
+<b>Values: </b><code>c++98, c++03, c++11, c++1z, c++14</code></br>
+Change the standard version used when building the tests.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">use_sanitizer=&lt;sanitizer name&gt;</h3>
+<blockquote class="lit-option-desc">
+<b>Values: </b><code>Memory, MemoryWithOrigins, Address, Undefined, None</code></br>
+Run the tests using the given sanitizer. If <code>LLVM_USE_SANITIZER</code>
+was given when building libc++ then that sanitizer will be used by default.
+If <code>None</code> is given then no sanitizer will be used.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">enable_warnings[=&lt;bool&gt;]</h3>
+<blockquote class="lit-option-desc">
+<b>Default: </b><code>False</code><br>
+Enable <code>-Wall -Wextra -pedantic</code> as well as disabling
+the system header pragmas.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">use_clang_verify[=&lt;bool&gt;]</h3>
+<blockquote class="lit-option-desc">
+<b>Default: </b><code>False</code><br>
+Enable the use of clang verif on tests that expect a compilation error.
+This verifies that the test fails to compile for the right reason.
+</blockquote>
+</p>
+
+<h2>Scan Build Options</h2>
+
+<p>
+<h3 class="lit-option">use_scan_build[=&lt;path/to/scan-build&gt;]</h3>
+<blockquote class="lit-option-desc">
+When specified the tests are compiled through scan-build. The optional value is
+used to specify the scan-build executable to use. If scan-build finds a defect
+the test will fail.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">scan_build_output=&lt;path/to/scan-build/output&gt;</h3>
+<blockquote class="lit-option-desc">
+Specify a folder in which to store the scan-build output. If scan-build is
+used without this option a temporary directory is created and reported at the
+start of the LIT run.
+</blockquote>
+</p>
+
+<p>
+<h3 class="lit-option">scan_build_args="&lt;list-of-args&gt;"</h3>
+<blockquote class="lit-option-desc">
+<b>Default: </b><code>-analyze-headers</code><br>
+Specify a list of args to be passed to <code>scan-build</code>.
+</blockquote>
+</p>
+
+<h2>Clang Tidy Options</h2>
+
+<p>
+<h3 class="lit-option">use_clang_tidy[=&lt;path/to/clang-tidy&gt;</h3>
+<blockquote class="lit-option_desc">
+When specified the tests are run through clang-tidy as an extra step. The test
+will fail if clang-tidy produces any output. The option value is used to specify
+the clang-tidy executable to use.
+</blockquote>
+</p>
+
+
+<p>
+<h3 class="lit-option">clang_tidy_args="&lt;list-of-args&gt;"</h3>
+<blockquote class="lit-option_desc">
+<b>Default: </b><code>-checks=-*,llvm*,clang-analyzer*,-clang-analyzer-alpha*
+                      -header-filter='.*'</code><br>
+Specify a list of args to be passed to <code>clang-tidy</code>.
+</blockquote>
+</p>
+
+
+</div>
+</body>
+</html>