[test-suite] Introduce a new CMake+LIT based harness

[jmolloy]

Summary

This introduces a replacement for the Makefile based build system. This replacement build system uses CMake for configuring and compiling the tests, and LIT for running them. LIT will report both compile and execution times as metrics, available by using the '-o' option to output JSON or '-v' (without '-s').

The advantages of this setup are:

• Use of the same technologies used in LLVM itself means there's more expertise available to debug, fix and extend, and fewer disparate command line flags to remember.
• A common-off-the-shelf solution means there's much less custom code to maintain, only glue.
• More readable output by default, more debuggable by far.
• The ability to run just one test, quickly and easily.
• The ability to perform test repeats without cleaning and recompiling first.
• Because the command line is simple, LNT is not required to run a test.

The disadvantages are:

• Not all of the features in the Makefile system are supported or are going to be. For example, llvm-gcc support, compiling with opt & llc. Clang is stable and expected now - we don't need these workarounds.

This commit introduces the CMake and LIT glue and a python script that can parse all the Makefiles in SingleSource & MultiSource and can generate equivalent CMakeLists.txt. The intention is that this is run once and the resulting files checked in then never used again.

With this script run, the entire test-suite builds and runs. Example command lines:

$ mkdir build; cd build
$ cmake -DCMAKE_C_COMPILER=/path/to/my/clang .. # CMake finds clang++ and introspects the target OS/arch/endianness
$ make -j12
$ llvm-lit -sv .

Ninja also works, but the way the ninja makefiles are generated means there is no way to build all tests in an arbitrary subtree (as LNT's --only-test does). With ninja you can build all tests or a single test.

All feedback welcome! Please also see the discussion on llvm-commits.

[beanz]

Thank you for doing this! This looks like a great start to get the ball rolling!

CMakeLists.txt

2

Do we need to make this 2.8.12.2? If we make it 3.2 then we can fold away some of the later version checks and features, and since this is a new CMake build system I'd kinda like to have it ditch support for legacy versions.

cmake/modules/SingleMultiSource.cmake

162

Probably need to add something like this add_dependencies(${source_exename} timeit fpcmp) here.

190

Same as above.

[MatzeB]

First: I really like where this is going! Thanks for working on this!

Some early comments after playing around with it for a while:

• I could not get lit to show the commandlines of tests that do not fail. This would be very nice to have for the benchmarks.
• I think we need cmake to generate a lit.cfg in the build directory or it will fail when the builddirectory is not a subdirectory of the test-suite.

I'd like to go further in simplifying the test commandlines. Right now the .test files still contain all the RunSafely.sh/timeit/... stuff. IMO this should be part of the benchmark driver (=lit.cfg). I think we should allow "RUN: " lines for the benchmark and "VERIFY:" lines for checking the benchmark results. With a custom lit test we should be able to prefix the RunSafely stuff automatically for all RUN: lines and run the VERIFY: lines as is. This is probably also useful if we are going to make the benchmark driver smarter and allow it for example to run a benchmark multiple times and aggregate the times :) In fact I started hacking on this and may have a proof of concept for this in the next days...

[asl]

CMakeLists.txt

2

I guess the real case is what are the "default" versions available on different systems we're going to support.

[jmolloy]

Hi guys,

Thanks for the quick feedback! Chris, I've added those dependencies. I still need to work out exactly what to do with timeit versus timeit.sh and add logic for switching between one or the other, although Mattias' patch in D14106 supersedes this.

I've removed the python script as Renato has kindly run it to generate the CMakeLists.txt and will check that in when this one is approved.

If you're happy, shall we get this in tree?

Cheers,

James

[jmolloy]

CMakeLists.txt

2

My unscientific straw poll of LLVM users inside ARM says that 2.8.12.2 is still the minimum we can expect - Ubuntu 14.10 LTS still ships this, I'm afraid, and as it's the latest LTS I'm not sure it's old enough to deprecate.

[jmolloy]

I could not get lit to show the commandlines of tests that do not fail. This would be very nice to have for the benchmarks.

This has been a feature request in the past for LIT, and honestly I thought it got implemented. A grep in my inbox turns up nothing though, so maybe I was delusional. Either way, it seems like a good feature to add to LIT and we should do that.

James

[beanz]

LGTM. One comment below, more for posterity than anything else.

CMakeLists.txt

2

The minimum version is dictated per-project, there is no global expectation. There is more burden involved in raising a version, than creating a new project with a higher minimum requirement, which is why I asked that we consider moving this up (to avoid having to move it up later).

We don't have any requirement or expectation that we will support the version of tools in an LTS Linux release. We have had this conversation numerous times, and it is not considered unreasonable that we might in the future request that Linux users behave like all the other users of LLVM and acquire their own copies of CMake (neither OS X nor Windows ship CMake or package managers). The reason LLVM and Clang haven't raised their minimum versions is that we haven't had compelling reasons to justify the change.

I believe that starting a new build system free of version-based checks is compelling reason to start this new project with a higher minimum version. Ideally I would want this to require at least CMake 3.2 which has been out for 6 months or so. That said, my personal preference for trying to push forward our CMake version isn't motivation enough to argue about landing this patch.

[tra]

cmake/modules/SingleMultiSource.cmake

139

When test suite is configured as part of whole llvm+projects+clang tree CMAKE_SOURCE_DIR points to the top of the tree and there's no lit-test-template.in there.

Perhaps you should use PROJECT_SOURCE_DIR here.

[beanz]

cmake/modules/SingleMultiSource.cmake

139

We should actually probably put a check in prohibiting building the test-suite in-tree. It doesn't work properly if you build it in-tree because it will use the system compiler not the just-built clang.

[tra]

CMakeLists.txt

68

Timeit ends up being $CMAKE_BINARY_DIR/bin/timeit if test-suite is configured as part of llvm tree.

[MatzeB]

Timeit ends up being $CMAKE_BINARY_DIR/bin/timeit if test-suite is configured as part of llvm tree.

Note that I will soon commit http://reviews.llvm.org/D14106 which will make this bit obsolete (as lit will insert the timeit/RunSafely calls then).

Disregard that, I was thinking about the timeit reference in the .test template file not this one.

CMakeLists.txt

+cmake_minimum_required(VERSION 2.8.12.2)
+
+# The test-suite is designed to be built in release mode anyway and
+# falls over unless -DNDEBUG is set.
+if (NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES)
+  message(STATUS "No build type selected, defaulting to Release")
+  set(CMAKE_BUILD_TYPE "Release")
+endif()
+
+add_definitions(-DNDEBUG)
+
+# Let's make sure lit output is pretty.
+if(CMAKE_VERSION VERSION_LESS 3.1.20141117)
+  set(cmake_3_2_USES_TERMINAL)
+else()
+  set(cmake_3_2_USES_TERMINAL USES_TERMINAL)
+endif()
+
+project(test-suite C CXX)
+
+# Add path for custom modules
+set(CMAKE_MODULE_PATH
+  ${CMAKE_MODULE_PATH}
+  "${CMAKE_CURRENT_SOURCE_DIR}/cmake"
+  "${CMAKE_CURRENT_SOURCE_DIR}/cmake/modules"
+  )
+
+# Sanity check our source directory to make sure that we are not trying to
+# generate an in-tree build (unless on MSVC_IDE, where it is ok), and to make
+# sure that we don't have any stray generated files lying around in the tree
+# (which would end up getting picked up by header search, instead of the correct
+# versions).
+if( CMAKE_SOURCE_DIR STREQUAL CMAKE_BINARY_DIR AND NOT MSVC_IDE )
+  message(FATAL_ERROR "In-source builds are not allowed.
+CMake would overwrite the makefiles distributed with LLVM.
+Please create a directory and run cmake from there, passing the path
+to this source directory as the last argument.
+This process created the file `CMakeCache.txt' and the directory `CMakeFiles'.
+Please delete them.")
+endif()
+
+include(MakefileFunctions)
+include(SingleMultiSource)
+find_package(YACC)
+find_package(TCL)
+
+if(NOT DEFINED TARGET_OS)
+  message(STATUS "Check target operating system - ${CMAKE_SYSTEM_NAME}")
+  set(TARGET_OS ${CMAKE_SYSTEM_NAME})
+endif()
+if(NOT DEFINED ARCH)
+  include(DetectArchitecture)
+  detect_architecture(ARCH)
+endif()
+if(NOT DEFINED ENDIAN)
+  include(TestBigEndian)
+  test_big_endian(IS_BIGENDIAN)
+  if(IS_BIGENDIAN)
+    set(ENDIAN "big")
+  else()
+    set(ENDIAN "little")
+  endif()
+endif()
+
+add_subdirectory(tools)
+
+# Now that the tools have been created, use tools/timeit to time all other compilations.
+set(CMAKE_C_COMPILE_OBJECT "${CMAKE_BINARY_DIR}/tools/timeit --summary <OBJECT>.time ${CMAKE_C_COMPILE_OBJECT}")
+set(CMAKE_CXX_COMPILE_OBJECT "${CMAKE_BINARY_DIR}/tools/timeit --summary <OBJECT>.time ${CMAKE_CXX_COMPILE_OBJECT}")
+
+add_subdirectory(SingleSource)
+add_subdirectory(MultiSource)

cmake/lit-test-template.in

+; RUN: ${RUNUNDER} ${CMAKE_SOURCE_DIR}/RunSafely.sh -t ${TIMEIT} 7200 ${STDIN_FILENAME} %t ${CMAKE_CURRENT_BINARY_DIR}/${exename} ${RUN_OPTIONS}
+; RUN: ${PROGRAM_OUTPUT_FILTER} %t
+; RUN: ${DIFFPROG} %t ${REFERENCE_OUTPUT}

cmake/modules/DetectArchitecture.c

+#if defined(__aarch64__)
+const char *str = "ARCHITECTURE IS AArch64";
+#elif defined(__arm__)
+const char *str = "ARCHITECTURE IS ARM";
+#elif defined(__alpha__)
+const char *str = "ARCHITECTURE IS Alpha";
+#elif defined(__mips__)
+const char *str = "ARCHITECTURE IS Mips";
+#elif defined(__powerpc__) || defined(__ppc__) || defined(__power__)
+const char *str = "ARCHITECTURE IS PowerPC";
+#elif defined(__sparc__)
+const char *str = "ARCHITECTURE IS Sparc";
+#elif defined(__xcore__)
+const char *str = "ARCHITECTURE IS XCore";
+#elif defined(__i386__) || defined(__x86_64__)
+const char *str = "ARCHITECTURE IS x86";
+#endif
+
+int main(int argc, char **argv) {
+    int require = str[argc];
+    (void)argv;
+    return require;
+}

cmake/modules/DetectArchitecture.cmake

+##===- DetectArchitecture.cmake -------------------------------------------===##
+#
+# Performs a try_compile to determine the architecture of the target.
+#
+##===----------------------------------------------------------------------===##
+
+macro(detect_architecture variable)
+  try_compile(HAVE_${variable}
+    ${CMAKE_BINARY_DIR}
+    ${CMAKE_SOURCE_DIR}/cmake/modules/DetectArchitecture.c
+    OUTPUT_VARIABLE OUTPUT
+    COPY_FILE ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/DetectArchitecture.bin)
+
+  if(HAVE_${variable})
+    file(STRINGS ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/DetectArchitecture.bin
+      DETECT_ARCH_STRING LIMIT_COUNT 1 REGEX "ARCHITECTURE IS")
+    if(DETECT_ARCH_STRING)
+      string(REGEX MATCH "[^ ]*$" DETECT_ARCH_MATCH ${DETECT_ARCH_STRING})
+      if(DETECT_ARCH_MATCH)
+        message(STATUS "Check target system architecture: ${DETECT_ARCH_MATCH}")
+        set(${variable} ${DETECT_ARCH_MATCH})
+      else()
+        message(SEND_ERROR "Could not detect target system architecture!")
+      endif()
+    else()
+      message(SEND_ERROR "Could not detect target system architecture!")
+    endif()
+  else()
+    message(STATUS "Determine the system architecture - failed")
+    file(APPEND ${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeError.log
+      "Determining the system architecture fialed with the following output:\n${OUTPUT}")
+    set(${variable})
+  endif()
+  
+endmacro(detect_architecture)
 No newline at end of file

cmake/modules/FindYACC.cmake

+##===- FindYACC.cmake -----------------------------------------------------===##
+#
+# Defines custom rules for building a YACC parser. Based on FindBISON.cmake.
+#
+##===----------------------------------------------------------------------===##
+
+find_program(YACC_EXECUTABLE yacc DOC "Path to the YACC executable")
+mark_as_advanced(YACC_EXECUTABLE)
+
+if(YACC_EXECUTABLE)
+
+  macro(add_yacc_parser target_name input_grammar output_c output_h)
+    add_custom_command(OUTPUT ${output_c} ${output_h}
+      COMMAND ${YACC_EXECUTABLE}
+      ARGS -d -o ${output_c} ${input_grammar}
+      DEPENDS ${input_grammar}
+      COMMENT "Building YACC parser ${output_c}"
+      WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
+  endmacro(add_yacc_parser)
+
+endif(YACC_EXECUTABLE)
+
+include(FindPackageHandleStandardArgs)
+find_package_handle_standard_args(YACC DEFAULT_MSG YACC_EXECUTABLE)

cmake/modules/MakefileFunctions.cmake

+##===- MakefileFunctions.cmake --------------------------------------------===##
+#
+# Defines functions used within subdirectory CMakeLists.txt, that look a bit
+# like standard GNU Make functions. These support the auto-generation of
+# CMakeLists.txt from Makefiles, although any use of these should probably be
+# looked at and manually translated into more idiomatic CMake.
+#
+##===----------------------------------------------------------------------===##
+
+# llvm_add_subdirectories - wrapper around add_subdirectory that can take
+# multiple items.
+macro(llvm_add_subdirectories)
+  foreach(V ${ARGV})
+    add_subdirectory(${V})
+  endforeach(V)
+endmacro()
+
+# llvm_prepend - equivalent to $(addprefix var,prefix). Modifies var in place.
+function(llvm_prepend var prefix)
+  set(listVar "")
+  foreach(f ${ARGN})
+    list(APPEND listVar "${prefix}${f}")
+  endforeach(f)
+  set(${var} "${listVar}" PARENT_SCOPE)
+endfunction()
+
+# llvm_filter - equivalent to $(filter needle,haystack). Returns a list containing
+# all of the items in ${ARGN} (after needle) that match needle.
+macro(llvm_filter output needle)
+  set(${output})
+  foreach(haystack_item ${ARGN})
+    foreach(needle_item ${needle})
+      if("${haystack_item}" STREQUAL "${needle_item}")
+        list(APPEND ${output} ${haystack_item})
+      endif()
+    endforeach()
+  endforeach()
+endmacro()
+
+# llvm_filter_out - equivalent to $(filter-out needle,haystack). Inverse of
+# llvm_filter.
+macro(llvm_filter_out output needle)
+  set(${output} "${ARGN}")
+  llvm_filter(tmp_output ${needle} ${ARGN})
+  list(REMOVE_ITEM ${output} ${tmp_output})
+endmacro()

cmake/modules/SingleMultiSource.cmake

+##===- SingleMultiSource.cmake --------------------------------------------===##
+#
+# Defines helpers to add executables and tests. The entry points to this
+# file are:
+#   `llvm_singlesource()` and
+#   `llvm_multisource()`
+#
+# Each is a macro that uses the environment it was called in to determine
+# what to build and how, and generates a test file that can be given to LIT.
+# The test file is generated at configure time.
+#
+##===----------------------------------------------------------------------===##
+
+# get_unique_exe_name - Given a source file name after which a test should be
+# named, create a unique name for the test. Usually this is just the source file
+# with the suffix stripped, but in some cases this ends up causing duplicates
+# so attempt to make each unique (by adding pathname segments until they become
+# unique).
+#
+# FIXME: Swap this with a simpler procedure to just append a numeral
+set_property(GLOBAL PROPERTY registered_executables)
+function(get_unique_exe_name new_name main_src)
+  get_property(registered_executables GLOBAL PROPERTY registered_executables)
+
+  string(REGEX REPLACE ".[cp]+$" "" path ${main_src})
+  string(REGEX REPLACE ".*/" "" name ${path})
+  list(FIND registered_executables ${name} name_idx)
+
+  if(${name_idx} EQUAL -1)
+    set(${new_name} ${name} PARENT_SCOPE)
+    set_property(GLOBAL APPEND PROPERTY registered_executables ${name})
+    return()
+  endif()
+
+  # There is a clash. Rename the target. Each time around the loop pull in
+  # a new path component.
+  foreach(n RANGE 1 4)
+    string(REGEX REPLACE ".*/([^/]+/${name})" "\\1" name ${path})
+    string(REGEX REPLACE "/" "-" safe_name ${name})
+
+    list(FIND registered_executables ${safe_name} name_idx)
+    if(${name_idx} EQUAL -1)
+      set(${new_name} ${safe_name} PARENT_SCOPE)
+      set_property(GLOBAL APPEND PROPERTY registered_executables ${safe_name})
+      return()
+    endif()
+  endforeach()
+  message(FATAL_ERROR "Failed to uniquify executable name!")
+endfunction()
+
+# append_cflags - add flags to the CFLAGS for target.
+macro(append_cflags target flags)
+  if(NOT "${${flags}}" STREQUAL "")
+    get_target_property(old_cflags ${target} COMPILE_FLAGS)
+    if(${old_cflags} STREQUAL "old_cflags-NOTFOUND")
+      set(old_cflags)
+    endif()
+    string(REPLACE ";" " " s "${old_cflags};${${flags}}")
+    set_target_properties(${target} PROPERTIES COMPILE_FLAGS ${s})
+  endif()
+endmacro()
+
+# append_ldflags - add flags to the LDFLAGS for target.
+macro(append_ldflags target flags)
+  if(NOT "${${flags}}" STREQUAL "")
+    get_target_property(old_ldflags ${target} LINK_FLAGS)
+    if(${old_ldflags} STREQUAL "old_ldflags-NOTFOUND")
+      set(old_ldflags)
+    endif()
+    string(REPLACE ";" " " s "${old_ldflags};${${flags}}")
+    set_target_properties(${target} PROPERTIES LINK_FLAGS ${s})
+  endif()
+endmacro()
+
+# llvm_add_test - Create a .test driver file suitable for LIT.
+#
+# The test template lives in cmake/lit-test-template.in and is configured by this function.
+function(llvm_add_test name exename)
+  if(NOT DEFINED STDIN_FILENAME)
+    set(STDIN_FILENAME /dev/null)
+  endif()
+
+  # Hash if we've been asked to, otherwise just use "touch" as an identity function.
+  if(HASH_PROGRAM_OUTPUT)
+    set(PROGRAM_OUTPUT_FILTER ${CMAKE_SOURCE_DIR}/HashProgramOutput.sh)
+  else()
+    set(PROGRAM_OUTPUT_FILTER touch)
+  endif()
+
+  # Find the reference output file key name.
+  if(SMALL_PROBLEM_SIZE)
+    set(KEY small)
+  elseif(LARGE_PROBLEM_SIZE)
+    set(KEY large)
+  else()
+    set(KEY)
+  endif()
+
+  # Pick the best reference output based on "programname.reference_output".
+  if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output.${ENDIAN}-endian.${KEY})
+    set(REFERENCE_OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output.${ENDIAN}-endian.${KEY})
+  elseif(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output.${KEY})
+    set(REFERENCE_OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output.${KEY})
+  elseif(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output.${ENDIAN}-endian)
+    set(REFERENCE_OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output.${ENDIAN}-endian)
+  elseif(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output)
+    set(REFERENCE_OUTPUT ${CMAKE_CURRENT_SOURCE_DIR}/${name}.reference_output)
+  else()
+    # Just compare to its own output. This will always succeed, but here's hoping the
+    # test in question uses its exit value to determine status, so it'll be caught by
+    # the previous RUN line.
+    set(REFERENCE_OUTPUT "%t")
+    message("-- No reference output found for test ${name}")
+  endif()
+
+  # If the program is nondeterministic, don't bother diffing and use "touch" again as an identity.
+  if(DEFINED PROGRAM_IS_NONDETERMINISTIC)
+    set(DIFFPROG touch)
+  else()
+    set(DIFFPROG ${CMAKE_BINARY_DIR}/tools/fpcmp)
+    if(DEFINED FP_TOLERANCE)
+      set(DIFFPROG "${DIFFPROG} -r ${FP_TOLERANCE}")
+    endif()
+    if(DEFINED FP_ABSTOLERANCE)
+      set(DIFFPROG "${DIFFPROG} -a ${FP_ABSTOLERANCE}")
+    endif()
+  endif()
+
+  if(DEFINED RUN_OPTIONS)
+    # RUN_OPTIONS is a semicolon-separated list. Change it into a whitespace-separated string.
+    string(REPLACE ";" " " RUN_OPTIONS "${RUN_OPTIONS}")
+  endif()
+
+  # FIXME: Sort out how to call timeit, or timeit.sh.
+  set(TIMEIT ${CMAKE_BINARY_DIR}/tools/timeit)
+  # FIXME: Add runtimelimit support.
+  
+  # Now do the actual configuration.
+  configure_file(${CMAKE_SOURCE_DIR}/cmake/lit-test-template.in
+    ${CMAKE_CURRENT_BINARY_DIR}/${exename}.test)
+endfunction()
+
+# llvm_singlesource - configure the current directory as a SingleSource subdirectory -
+# i.e. every file in *.{c,cpp,cc} is treated as its own test.
+macro(llvm_singlesource)
+  file(GLOB sources *.c *.cpp *.cc)
+  foreach(source ${sources})
+    # Find the pure name of the test
+    string(REGEX REPLACE ".[cp]+$" "" path ${source})
+    string(REGEX REPLACE ".*/" "" name ${path})
+
+    # Should we skip this?
+    list(FIND PROGRAMS_TO_SKIP ${name} name_idx)
+    if(${name_idx} EQUAL -1)
+      get_unique_exe_name(source_exename ${source})
+      add_executable(${source_exename} ${source})
+      append_cflags(${source_exename} CFLAGS)
+      append_cflags(${source_exename} CPPFLAGS)
+      append_cflags(${source_exename} CXXFLAGS)
+      append_ldflags(${source_exename} LDFLAGS)
+      llvm_add_test(${name} ${source_exename})
+      add_dependencies(${source_exename} timeit fpcmp)
+    endif()
+  endforeach()
+endmacro()
+
+# llvm_multisource - configure the current directory as a MultiSource subdirectory -
+# i.e. there is one test and it consists of all sources in the directory (or a curated
+# list, if Source is defined).
+macro(llvm_multisource)
+  if(DEFINED Source)
+    set(sources ${Source})
+  else()
+    file(GLOB sources *.c *.cpp *.cc)
+  endif()
+  list(LENGTH sources sources_len)
+
+  if(sources_len GREATER 0 AND DEFINED PROG)
+  # Should we skip this?
+  list(FIND PROGRAMS_TO_SKIP ${PROG} name_idx)
+  if(${name_idx} EQUAL -1)
+    include_directories(${CMAKE_CURRENT_SOURCE_DIR})
+    include_directories(${CMAKE_CURRENT_BINARY_DIR})
+    get_unique_exe_name(source_exename "${PROG}.c")
+    add_executable(${source_exename} ${sources})
+    append_cflags(${source_exename} CFLAGS)
+    append_cflags(${source_exename} CPPFLAGS)
+    append_cflags(${source_exename} CXXFLAGS)
+    append_ldflags(${source_exename} LDFLAGS)
+    llvm_add_test(${PROG} ${source_exename})
+    add_dependencies(${source_exename} timeit fpcmp)
+  endif()
+endif()
+endmacro()

lit.cfg

+import lit.formats
+import lit
+import os, glob, re
+
+def getUserTimeFromTimeOutput(f):
+    with open(f) as fd:
+        l = [l for l in fd.readlines()
+             if l.startswith('user')]
+    assert len(l) == 1
+        
+    m = re.match(r'user\s+([0-9.]+)', l[0])
+    return float(m.group(1))
+
+class TestSuiteTest(lit.formats.ShTest):
+    def __init__(self):
+        lit.formats.ShTest.__init__(self, False)
+        
+    def execute(self, test, litConfig):
+        result = lit.formats.ShTest.execute(self, test, litConfig)
+        basepath = os.path.dirname(test.getFilePath())
+        
+        if not result.code.isFailure:
+            # Collect the timing information.
+            timeglob = os.path.join(basepath, 'Output', '*.time')
+            times = glob.glob(timeglob)
+            assert len(times) == 1
+            time = getUserTimeFromTimeOutput(times[0])
+
+            result.addMetric('exec_time', lit.Test.toMetricValue(time))
+
+        # For completeness, attempt to find compile time information too.
+        compile_time = 0.0
+        for path, subdirs, files in os.walk(basepath):
+            for file in files:
+                if file.endswith('.o.time'):
+                    compile_time += getUserTimeFromTimeOutput(os.path.join(path, file))
+        result.addMetric('compile_time', lit.Test.toMetricValue(compile_time))
+        
+        return result
+
+config.name = 'test-suite'
+
+config.test_format = TestSuiteTest()
+config.suffixes = ['.test']
+config.test_source_root = os.path.dirname(__file__)
+config.excludes = ['ABI-Testsuite']

tools/CMakeLists.txt

+
+# FIXME: These need to be host-compiled, if we're cross compiling.
+# FIXME: Replicate Makefile.tools's logic for determining whether to use fpcmp/fpcmp.sh
+add_executable(fpcmp fpcmp.c)
+add_executable(timeit timeit.c)