[LNT] Add cross-compilation support to 'lnt runtest test-suite'

[kristof.beyls]

Summary

This adds cross-compilation support for the cmake+lit-based way to build and run the test-suite, by letting users specify a cmake toolchain file if they want to cross-compile.

The advantage of doing so over other ways to enable cross-compiling is:

• It is in line with the recommended cmake way to cross-compile a project: using a toolchain file.
• Fewer command line options need to be given to lnt runtest test-suite.

Regression tests and documentation still needs to be written for this, but I first wanted to get feedback on this approach before spending more time on it.

FWIW, an actual lnt runtest test-suite invoke command that I use to cross-compile and cross-test the test-suite on for an aarch64-target is:

lnt runtest test-suite \
  --test-suite ~/dev/llvm.org/test-suite -j40 \
  --cppflags="-O3" \
  --run-under=$HOME/dev/aarch64-emu/aarch64-qemu.sh \
  --use-lit ~/dev/llvm.org/build/bin/llvm-lit \
  --cross-compiling-toolchain-file=$HOME/dev/llvm.org/clang_aarch64_linux.cmake

With the content of clang_aarch64_linux.cmake being:

set(CMAKE_SYSTEM_NAME Linux )
set(triple aarch64-linux-gnu )
set(CMAKE_C_COMPILER /home/kribey01/dev/llvm.org/build/bin/clang )
set(CMAKE_C_COMPILER_TARGET ${triple} )
set(CMAKE_CXX_COMPILER /home/kribey01/dev/llvm.org/build/bin/clang++ )
set(CMAKE_CXX_COMPILER_TARGET ${triple} )
set(CMAKE_SYSROOT /home/kribey01/dev/aarch64-emu/sysroot-glibc-linaro-2.23-2016.11-aarch64-linux-gnu )
set(CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN /home/kribey01/dev/aarch64-emu/gcc-linaro-6.2.1-2016.11-x86_64_aarch64-linux-gnu)
set(CMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN /home/kribey01/dev/aarch64-emu/gcc-linaro-6.2.1-2016.11-x86_64_aarch64-linux-gnu)

I intend to put the above details in the documentation, if we decide this is the right approach to take.

What do you think?

[kristof.beyls]

lnt/tests/test_suite.py

184 ↗

(On Diff #85387)

This probably should pick up opts.cmake, not the cmake which happens to be on the path.

[rengolin]

Hi Kristof,

Thanks for working on this, we really needed this functionality.

My question is if we shouldn't have only one way to pass options. I'm worried that the external file will confuse people.

For instance, if you specify the cross compiler by full path, it should be able to figure out the sysroot (bin/..) in most cases.

Also, that would work for GCC (with the triple in the name), but not for Clang. I see you're using "TARGET" for that, but other options (like -mthumb -mfloat-abi etc) will need to be passed as well.

I think these can be reduced cleanly to:

• CROSS_CC / CROSS_CXX
• CROSS_CFLAGS / CROSS_CXXFLAGS

and thus passed via LNT -- flags like all the others.

cheers,
--renato

lnt/tests/test_suite.py

389 ↗

(On Diff #85387)

So the tools like fpcmp will be compiled to the target and executed on the QEMU as well?

[kristof.beyls]

In D29030#653374, @rengolin wrote:

Hi Kristof,

Thanks for working on this, we really needed this functionality.

My question is if we shouldn't have only one way to pass options. I'm worried that the external file will confuse people.

The external file seems to be the canonical way to cross-compile cmake-based projects, e.g. see https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling.
Therefore, I think that following "the one true way to do cross-compiling, the cmake-approach" in the long run will lead to least confusion.

For instance, if you specify the cross compiler by full path, it should be able to figure out the sysroot (bin/..) in most cases.

Maybe in most cases, but at least not in the case of my setup, where I'm testing a clang I'm developing myself, with a Linaro-provided sysroot.
Anyway, I hope that by explicitly showing a cmake toolchain file in the documentation of LNT, it will help people to write a toolchain file. Although this really is cmake documentation (and the documentation at the URL I provided above wasn't too bad: it helped me to quickly write the clang-aarch64-linux.cmake file I've posted as an example in this thread above), I think it'd be useful to show a concrete example with a clang toolchain to help llvm developers to get up and running quickly.

Also, that would work for GCC (with the triple in the name), but not for Clang. I see you're using "TARGET" for that, but other options (like -mthumb -mfloat-abi etc) will need to be passed as well.

The other options are being passed via the lnt command line, using --cppflags. Also see how I pass "-O3" in the example I have above.
TARGET seems to be the cmake approach to specify the target for cross-compiling, also see the URL I pointed at above. Again, I'd like to stick with how cmake documents how to do cross-compilation.

I think these can be reduced cleanly to:

• CROSS_CC / CROSS_CXX
• CROSS_CFLAGS / CROSS_CXXFLAGS

and thus passed via LNT -- flags like all the others.

cheers,
--renato

lnt/tests/test_suite.py

389 ↗

(On Diff #85387)

No, the test-suite's cmake infrastructure already handles this well: tools like fpcmp are built and run natively on the host.

[rengolin]

In D29030#653394, @kristof.beyls wrote:

The external file seems to be the canonical way to cross-compile cmake-based projects, e.g. see https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling.
Therefore, I think that following "the one true way to do cross-compiling, the cmake-approach" in the long run will lead to least confusion.

I had no idea! :)

Sounds good.

[MatzeB]

What about the existing --cmake-cache flag? We are happily using this for crosscompiling since the early days.
You can find our toolchain files in test-suite/cmake/caches/target-*.cmake. I haven't used the cmake XXX_EXTERNAL_TOOLCHAIN and XXX_TARGET flags of cmake yet (as I believe they don't match how xcode toolchains work when crosscompiling) but hopefully they just work in your case.

Some tips on the cache file you proposed above:

• You most probably need set(Variable Value CACHE STRING "") to have any effects
• I introduced TEST_SUITE_ARCH_FLAGS for cases where you want to add crosscompilation related flags to all compiler and linker invocations. Having this in a separate variable is often useful as you don't compete with people passing in custom C/C++ flags and overriding the flags necessary for the cross compilation.

[MatzeB]

In D29030#653483, @MatzeB wrote:

What about the existing --cmake-cache flag? We are happily using this for crosscompiling since the early days.
You can find our toolchain files in test-suite/cmake/caches/target-*.cmake. I haven't used the cmake XXX_EXTERNAL_TOOLCHAIN and XXX_TARGET flags of cmake yet (as I believe they don't match how xcode toolchains work when crosscompiling) but hopefully they just work in your case.

Some tips on the cache file you proposed above:

• You most probably need set(Variable Value CACHE STRING "") to have any effects
• I introduced TEST_SUITE_ARCH_FLAGS for cases where you want to add crosscompilation related flags to all compiler and linker invocations. Having this in a separate variable is often useful as you don't compete with people passing in custom C/C++ flags and overriding the flags necessary for the cross compilation.

Reading the cmake docu I realize their toolchain files are something different than using cache files. So those two points are probably not relevant to you.

As for the rest of the commit: Using a toolchain file looks like just passing a -D flag which is already supported. As for picking up the compiler, couldn't you simply extract that from CMakeCache.txt? And if that doesn't work, shouldn't we rather upstream the changes to CMakeLists.txt that print the compiler instead of having LNT mess around with them?

[MatzeB]

lnt/tests/test_suite.py

389 ↗

(On Diff #85387)

qemu can probably be used with the --run-under flag.

[kristof.beyls]

In D29030#653494, @MatzeB wrote:

In D29030#653483, @MatzeB wrote:

What about the existing --cmake-cache flag? We are happily using this for crosscompiling since the early days.
You can find our toolchain files in test-suite/cmake/caches/target-*.cmake. I haven't used the cmake XXX_EXTERNAL_TOOLCHAIN and XXX_TARGET flags of cmake yet (as I believe they don't match how xcode toolchains work when crosscompiling) but hopefully they just work in your case.

Some tips on the cache file you proposed above:

• You most probably need set(Variable Value CACHE STRING "") to have any effects
• I introduced TEST_SUITE_ARCH_FLAGS for cases where you want to add crosscompilation related flags to all compiler and linker invocations. Having this in a separate variable is often useful as you don't compete with people passing in custom C/C++ flags and overriding the flags necessary for the cross compilation.

Reading the cmake docu I realize their toolchain files are something different than using cache files. So those two points are probably not relevant to you.

I haven't read/understood the --cmake-cache functionality yet; will look into that first.

As for the rest of the commit: Using a toolchain file looks like just passing a -D flag which is already supported. As for picking up the compiler, couldn't you simply extract that from CMakeCache.txt? And if that doesn't work, shouldn't we rather upstream the changes to CMakeLists.txt that print the compiler instead of having LNT mess around with them?

I'm trying to eliminate as many not-needed command line options as possible, e.g. --llvm-arch, --cross-compiling, ... . There are already more than enough command line options to lnt runtest, reducing the amount of them probably would make it simpler to use.
Indeed, just using a -D flag to enable cross-compiling instead of having a separate lnt runtest command line option for it would be preferable (and then documenting how to use it to do cross-compiling).
But then, doesn't the same argument apply to dropping lnt runtest -cflags or lnt runtest -cppflags: they also both could be set using -D?
Anyway, let's not go there as part of this patch; I'll first look into whether I can eliminate the need for opts.cc and opts.cxx also for the non-cross-compiling case. In that case, I indeed no longer see the need for a separate option so that lnt is aware we're cross-compiling, and instead we can just use -D.

lnt/tests/test_suite.py

389 ↗

(On Diff #85387)

Indeed, that's what I'm doing in the example 'lnt runtest test-suite' invocation command I gave above.

[MatzeB]

In D29030#654626, @kristof.beyls wrote:

In D29030#653494, @MatzeB wrote:

In D29030#653483, @MatzeB wrote:

What about the existing --cmake-cache flag? We are happily using this for crosscompiling since the early days.
You can find our toolchain files in test-suite/cmake/caches/target-*.cmake. I haven't used the cmake XXX_EXTERNAL_TOOLCHAIN and XXX_TARGET flags of cmake yet (as I believe they don't match how xcode toolchains work when crosscompiling) but hopefully they just work in your case.

Some tips on the cache file you proposed above:

• You most probably need set(Variable Value CACHE STRING "") to have any effects
• I introduced TEST_SUITE_ARCH_FLAGS for cases where you want to add crosscompilation related flags to all compiler and linker invocations. Having this in a separate variable is often useful as you don't compete with people passing in custom C/C++ flags and overriding the flags necessary for the cross compilation.

Reading the cmake docu I realize their toolchain files are something different than using cache files. So those two points are probably not relevant to you.

I haven't read/understood the --cmake-cache functionality yet; will look into that first.

cmake caches allow you to prepopulate the cache. In practice this allows you to specify a single cache file instead of passing a whole set of -D flags on the cmake commandline (and you can execute cmake code so you can query the environment and change flags based on that). I find a very convenient tool to bake a specific build/test configuration into a single file and share it.
From reading the cmake docu it sounds like the toolchain files are specifically designed to get crosscompilation in the typical gcc style right (i.e. calling target-triple-tool instead of tool), so this is probably what you want to use as the primary tool for your crosscompilation tasks (though if you need additional options you may still create a cache file that sets the toolchain and the additional options :)

As for the rest of the commit: Using a toolchain file looks like just passing a -D flag which is already supported. As for picking up the compiler, couldn't you simply extract that from CMakeCache.txt? And if that doesn't work, shouldn't we rather upstream the changes to CMakeLists.txt that print the compiler instead of having LNT mess around with them?

I'm trying to eliminate as many not-needed command line options as possible, e.g. --llvm-arch, --cross-compiling, ... . There are already more than enough command line options to lnt runtest, reducing the amount of them probably would make it simpler to use.
Indeed, just using a -D flag to enable cross-compiling instead of having a separate lnt runtest command line option for it would be preferable (and then documenting how to use it to do cross-compiling).
But then, doesn't the same argument apply to dropping lnt runtest -cflags or lnt runtest -cppflags: they also both could be set using -D?
Anyway, let's not go there as part of this patch; I'll first look into whether I can eliminate the need for opts.cc and opts.cxx also for the non-cross-compiling case. In that case, I indeed no longer see the need for a separate option so that lnt is aware we're cross-compiling, and instead we can just use -D.

My argument would be to drop the whole "lnt runtest" mode and let users do the cmake/ninja/lit sequence themselfes and teach "lnt submit" to just submit the results. That way you are never limited by missing or confusing options in lnt runtest, have one layer less to debug and we have one thing less to maintain and keep up to date. Though we may need to write more documentation about the cmake/lit test-suite.

[kristof.beyls]

As for the rest of the commit: Using a toolchain file looks like just passing a -D flag which is already supported. As for picking up the compiler, couldn't you simply extract that from CMakeCache.txt? And if that doesn't work, shouldn't we rather upstream the changes to CMakeLists.txt that print the compiler instead of having LNT mess around with them?

I'm trying to eliminate as many not-needed command line options as possible, e.g. --llvm-arch, --cross-compiling, ... . There are already more than enough command line options to lnt runtest, reducing the amount of them probably would make it simpler to use.
Indeed, just using a -D flag to enable cross-compiling instead of having a separate lnt runtest command line option for it would be preferable (and then documenting how to use it to do cross-compiling).
But then, doesn't the same argument apply to dropping lnt runtest -cflags or lnt runtest -cppflags: they also both could be set using -D?
Anyway, let's not go there as part of this patch; I'll first look into whether I can eliminate the need for opts.cc and opts.cxx also for the non-cross-compiling case. In that case, I indeed no longer see the need for a separate option so that lnt is aware we're cross-compiling, and instead we can just use -D.

My argument would be to drop the whole "lnt runtest" mode and let users do the cmake/ninja/lit sequence themselfes and teach "lnt submit" to just submit the results. That way you are never limited by missing or confusing options in lnt runtest, have one layer less to debug and we have one thing less to maintain and keep up to date. Though we may need to write more documentation about the cmake/lit test-suite.

Interesting idea, I like it! When I explain others (non-LLVM-developers) about what lnt is, the first thing I explain is that it is both a tool to run the LLVM test-suite (which mostly isn't interesting to the non-LLVM-developers) and a server/database/analysis engine to keep track of code generation quality of a code generator (which is interesting to non-LLVM code generator developers). At least from this point of view, it'd be nice if lnt was "one thing" rather than "two things".
Maybe 'lnt submit' could also be dropped, and users could upload the produced json after a lit run using e.g. curl? On one hand, submitting jsons to the server could be classified as being part of the "server/database/analysis engine" functionality. OTOH, it might remove the need for lnt on the client/test machine side completely?

[kristof.beyls]

Thanks for all the feedback and the interesting discussion!
I've adapted the patch accordingly, going for using cmake features more directly rather than adding ever more lnt runtest command line options.
I've also added documentation that should help with using 'lnt runtest test-suite' in a cross-compiling setup.

The main functional changes are:

• Making sure the correct target is picked up for the report.json.
• --cc is no longer a required command line option.
• Documentation is written to explain how to best perform test-suite runs in a cross-compile set up.

[kristof.beyls]

ping

[MatzeB]

lnt/tests/test_suite.py

760–763 ↗

(On Diff #88516)

Shouldn't we rather put the --target flag into the toolchain file, is that not possible?

If we need to do it from the lnt side anyway, I'd recommend setting TEST_SUITE_ARCH_FLAGS which is a convention I introduced to the CMakeLists.txt. These are inserted to all compiler and linker invocations (so do not depend on our current "misfeature" that cflags are passed to C compiler, C++ compiler and linker) and plays nicer with people who override the CFLAGS in cache files.

766 ↗

(On Diff #88516)

This name feels wrong to me as this does not return the cmake variables which would be CMAKE_C_COMPILER, CMAKE_BUILD_TYPE, ... but instead returns some LNT specific names like 'cc', 'build_type', ...

797 ↗

(On Diff #88516)

Looking at the earlier usage of these functions, _extract_cmake_vars_from_cache() is called once anyway while possible _get_cc_info() calls repeat this, maybe you better let the user feed in the cmake_vars information as an argument since it is available anyway.

[MatzeB]

In any way this LGTM (with the previous nitpicks addressed or explained).

[kristof.beyls]

lnt/tests/test_suite.py

760–763 ↗

(On Diff #88516)

Hi Matthias,

This _get_target_flags is used only when invoking lnt.testing.util.compilers.get_cc_info, see line 799 of this patch.
get_cc_info uses direct compiler invokes to find out information like what the target is to put into the metadata report.json.
Maybe we should let lnt.testing.util.compilers.get_cc_info also work with a cmake toolchain file, I'm not sure - but that definitly sounds something for an independent patch.
Probably the best thing to do for now is to get rid of the _get_target_flags function call and just inline the code into the one place where it's used: _get_cc_info.
So, in short, the cmake toolchain file does specify the target, but get_cc_info doesn't work with cmake toolchain files, just with command line arguments. This translates the target specified in the cmake toolchain file into what get_cc_info needs.

766 ↗

(On Diff #88516)

You're right, this should just return a dictionary with cmake variables, not LNT-specific names.

797 ↗

(On Diff #88516)

_get_cc_info is actually called from 2 places, and there's a bit of plumbing cmake_vars through function calls needed.
But all in all, the required level of plumbing through variables doesn't hit my personal threshold yet for not doing so, so OK, let's do it.

[MatzeB]

Thanks, LGTM still stands.

lnt/tests/test_suite.py

760–763 ↗

(On Diff #88516)

Ah ok, I missed that it is not self.cflags here, but something used locally. That's certainly ok for now.

Long term I agree that we should rather have cmake fill out the cc information and place that in a file or similar, but that is for another patch.

761–774 ↗

(On Diff #90705)

Maybe we should just extract them all instead of maintaining a whitelist. But I don't really care either way.

lnt/trunk/docs/tests.rst

$ lnt runtest test-suite \
          --use-lit=~/llvm/utils/lit/lit.py \
          --test-suite ~/llvm-test-suite \
          --cmake-cache Release \
          --diagnose --only-test SingleSource/Benchmarks/Stanford/Bubblesort
 
 This will run the test-suite many times over, collecting useful information
 in a report directory. The report collects many things like execution profiles,
 compiler time reports, intermediate files, binary files, and build information.
+
+
+Cross-compiling
++++++++++++++++
+
+The best way to run the test-suite in a cross-compiling setup with the
+cmake driver is to use cmake's built-in support for cross-compiling as much as
+possible. In practice, the recommended way to cross-compile is to use a cmake
+toolchain file (see
+https://cmake.org/cmake/help/v3.0/manual/cmake-toolchains.7.html#cross-compiling)
+
+An example command line for cross-compiling on an X86 machine, targeting
+AArch64 linux, is::
+
+  $ lnt runtest test-suite \
+         --sandbox SANDBOX \
+         --test-suite /work/llvm-test-suite \
+         --use-lit lit \
+         --cppflags="-O3" \
+         --run-under=$HOME/dev/aarch64-emu/aarch64-qemu.sh \
+         --cmake-define=CMAKE_TOOLCHAIN_FILE:FILEPATH=$HOME/clang_aarch64_linux.cmake
+
+The key part here is the CMAKE_TOOLCHAIN_FILE define. As you're
+cross-compiling, you may need a --run-under command as the produced binaries
+probably won't run natively on your development machine, but something extra
+needs to be done (e.g. running under a qemu simulator, or transferring the
+binaries to a development board). This isn't explained further here.
+
+In your toolchain file, it's important to specify that the cmake variables
+defining the toolchain must be cached in CMakeCache.txt, as that's where lnt
+reads them from to figure out which compiler was used when needing to construct
+metadata for the json report. An example is below. The important keywords to
+make the variables appear in the CMakeCache.txt are "CACHE STRING "" FORCE"::
+
+  $ cat clang_aarch64_linux.cmake
+  set(CMAKE_SYSTEM_NAME Linux )
+  set(triple aarch64-linux-gnu )
+  set(CMAKE_C_COMPILER /home/user/build/bin/clang CACHE STRING "" FORCE)
+  set(CMAKE_C_COMPILER_TARGET ${triple} CACHE STRING "" FORCE)
+  set(CMAKE_CXX_COMPILER /home/user/build/bin/clang++ CACHE STRING "" FORCE)
+  set(CMAKE_CXX_COMPILER_TARGET ${triple} CACHE STRING "" FORCE)
+  set(CMAKE_SYSROOT /home/user/aarch64-emu/sysroot-glibc-linaro-2.23-2016.11-aarch64-linux-gnu )
+  set(CMAKE_C_COMPILER_EXTERNAL_TOOLCHAIN /home/user/aarch64-emu/gcc-linaro-6.2.1-2016.11-x86_64_aarch64-linux-gnu )
+  set(CMAKE_CXX_COMPILER_EXTERNAL_TOOLCHAIN /home/user/aarch64-emu/gcc-linaro-6.2.1-2016.11-x86_64_aarch64-linux-gnu )
+
+

lnt/trunk/lnt/tests/test_suite.py

def run_test(self, name, args):
         group = OptionGroup(parser, "Test compiler")
         group.add_option("", "--cc", dest="cc", metavar="CC",
                          type=str, default=None,
                          help="Path to the C compiler to test")
         group.add_option("", "--cxx", dest="cxx", metavar="CXX",
                          type=str, default=None,
                          help="Path to the C++ compiler to test (inferred from"
                               " --cc where possible")
-        group.add_option("", "--llvm-arch", dest="llvm_arch",
-                         type='choice', default=None,
-                         help="Override the CMake-inferred architecture",
-                         choices=TEST_SUITE_KNOWN_ARCHITECTURES)
-        group.add_option("", "--cross-compiling", dest="cross_compiling",
-                         action="store_true", default=False,
-                         help="Inform CMake that it should be cross-compiling")
-        group.add_option("", "--cross-compiling-system-name", type=str,
-                         default=None, dest="cross_compiling_system_name",
-                         help="The parameter to pass to CMAKE_SYSTEM_NAME when"
-                              " cross-compiling. By default this is 'Linux' "
-                              "unless -arch is in the cflags, in which case "
-                              "it is 'Darwin'")
         group.add_option("", "--cppflags", type=str, action="append",
                          dest="cppflags", default=[],
                          help="Extra flags to pass the compiler in C or C++ mode. "
                               "Can be given multiple times")
         group.add_option("", "--cflags", type=str, action="append",
                          dest="cflags", default=[],
                          help="Extra CFLAGS to pass to the compiler. Can be "
                               "given multiple times")
def run_test(self, name, args):
 
         if len(args) == 0:
             self.nick = platform.uname()[1]
         elif len(args) == 1:
             self.nick = args[0]
         else:
             parser.error("Expected no positional arguments (got: %r)" % (args,))
 
-        for a in ['cross_compiling', 'cross_compiling_system_name', 'llvm_arch']:
-            if getattr(opts, a):
-                parser.error('option "%s" is not yet implemented!' % a)
-
         if self.opts.sandbox_path is None:
             parser.error('--sandbox is required')
 
-        if self.opts.cc is None:
-            parser.error('--cc is required')
-
-        # Option validation.
-        opts.cc = resolve_command_path(opts.cc)
+        if self.opts.cc is not None:
+            self.opts.cc = resolve_command_path(self.opts.cc)
 
-        if not lnt.testing.util.compilers.is_valid(opts.cc):
+            if not lnt.testing.util.compilers.is_valid(self.opts.cc):
                 parser.error('--cc does not point to a valid executable.')
 
             # If there was no --cxx given, attempt to infer it from the --cc.
-        if opts.cxx is None:
-            opts.cxx = lnt.testing.util.compilers.infer_cxx_compiler(opts.cc)
-            if opts.cxx is not None:
-                note("Inferred C++ compiler under test as: %r" % (opts.cxx,))
+            if self.opts.cxx is None:
+                self.opts.cxx = \
+                    lnt.testing.util.compilers.infer_cxx_compiler(self.opts.cc)
+                if self.opts.cxx is not None:
+                    note("Inferred C++ compiler under test as: %r"
+                         % (self.opts.cxx,))
                 else:
                     parser.error("unable to infer --cxx - set it manually.")
             else:
-            opts.cxx = resolve_command_path(opts.cxx)
+                self.opts.cxx = resolve_command_path(self.opts.cxx)
 
-        if not os.path.exists(opts.cxx):
-            parser.error("invalid --cxx argument %r, does not exist" % (opts.cxx))
+            if not os.path.exists(self.opts.cxx):
+                parser.error("invalid --cxx argument %r, does not exist"
+                             % (self.opts.cxx))
 
         if opts.test_suite_root is None:
             parser.error('--test-suite is required')
         if not os.path.exists(opts.test_suite_root):
             parser.error("invalid --test-suite argument, does not exist: %r" % (
                 opts.test_suite_root))
 
         if opts.test_suite_externals:
def run_test(self, name, args):
         #  If we are doing diagnostics, skip the usual run and do them now.
         if opts.diagnose:
             return self.diagnose()
 
         # configure, so we can extract toolchain information from the cmake
         # output.
         self._configure_if_needed()
 
+        # Verify that we can actually find a compiler before continuing
+        cmake_vars = self._extract_cmake_vars_from_cache()
+        if "CMAKE_C_COMPILER" not in cmake_vars or \
+                not os.path.exists(cmake_vars["CMAKE_C_COMPILER"]):
+            parser.error(
+                "Couldn't find C compiler (%s). Maybe you should specify --cc?"
+                % cmake_vars.get("CMAKE_C_COMPILER"))
+
         # We don't support compiling without testing as we can't get compile-
         # time numbers from LIT without running the tests.
         if opts.compile_multisample > opts.exec_multisample:
             note("Increasing number of execution samples to %d" %
                  opts.compile_multisample)
             opts.exec_multisample = opts.compile_multisample
 
         if opts.auto_name:
             # Construct the nickname from a few key parameters.
-            cc_info = self._get_cc_info()
+            cc_info = self._get_cc_info(cmake_vars)
             cc_nick = '%s_%s' % (cc_info['cc_name'], cc_info['cc_build'])
             self.nick += "__%s__%s" % (cc_nick,
                                        cc_info['cc_target'].split('-')[0])
         note('Using nickname: %r' % self.nick)
 
         #  When we can't detect the clang version we use 0 instead. That
         # is a horrible failure mode because all of our data ends up going
         # to order 0.  The user needs to give an order if we can't detect!
         if opts.run_order is None:
-            cc_info = self._get_cc_info()
+            cc_info = self._get_cc_info(cmake_vars)
             if cc_info['inferred_run_order'] == 0:
                 fatal("Cannot detect compiler version. Specify --run-order"
                       " to manually define it.")
 
         # Now do the actual run.
         reports = []
         json_reports = []
         for i in range(max(opts.exec_multisample, opts.compile_multisample)):
             c = i < opts.compile_multisample
             e = i < opts.exec_multisample
-            run_report, json_data = self.run(compile=c, test=e)
+            run_report, json_data = self.run(cmake_vars, compile=c, test=e)
             reports.append(run_report)
             json_reports.append(json_data)
 
         report = self._create_merged_report(reports)
 
         # Write the report out so it can be read by the submission tool.
         report_path = os.path.join(self._base_path, 'report.json')
         with open(report_path, 'w') as fd:
class TestSuiteTest(BuiltinTest):
 
     def _configure_if_needed(self):
         mkdir_p(self._base_path)
         if not self.configured:
             self._configure(self._base_path)
             self._clean(self._base_path)
             self.configured = True
 
-    def run(self, compile=True, test=True):
+    def run(self, cmake_vars, compile=True, test=True):
         mkdir_p(self._base_path)
 
         if self.opts.pgo:
             self._collect_pgo(self._base_path)
             self.trained = True
             self.configured = False
 
         self._configure_if_needed()
 
         if self.compiled and compile:
             self._clean(self._base_path)
         if not self.compiled or compile:
             self._make(self._base_path)
             self.compiled = True
 
         data = self._lit(self._base_path, test)
-        return self._parse_lit_output(self._base_path, data), data
+        return self._parse_lit_output(self._base_path, data, cmake_vars), data
 
     def _create_merged_report(self, reports):
         if len(reports) == 1:
             return reports[0]
 
         machine = reports[0].machine
         run = reports[0].run
         run.end_time = reports[-1].run.end_time
def _clean(self, path):
             subdir = os.path.join(*components)
 
         self._check_call([make_cmd, 'clean'],
                          cwd=subdir)
 
     def _configure(self, path, extra_cmake_defs=[], execute=True):
         cmake_cmd = self.opts.cmake
 
-        defs = {
-            # FIXME: Support ARCH, SMALL/LARGE etc
-            'CMAKE_C_COMPILER': self.opts.cc,
-            'CMAKE_CXX_COMPILER': self.opts.cxx,
-        }
+        defs = {}
+        if self.opts.cc:
+            defs['CMAKE_C_COMPILER'] = self.opts.cc
+        if self.opts.cxx:
+            defs['CMAKE_CXX_COMPILER'] = self.opts.cxx,
+
         if self.opts.cppflags or self.opts.cflags:
             all_cflags = ' '.join([self.opts.cppflags, self.opts.cflags])
             defs['CMAKE_C_FLAGS'] = self._unix_quote_args(all_cflags)
 
         if self.opts.cppflags or self.opts.cxxflags:
             all_cxx_flags = ' '.join([self.opts.cppflags, self.opts.cxxflags])
             defs['CMAKE_CXX_FLAGS'] = self._unix_quote_args(all_cxx_flags)
 
def _get_lnt_code(self, code):
                 'UNRESOLVED': lnt.testing.FAIL
                 }[code]
 
     def _test_failed_to_compile(self, raw_name, path):
         # FIXME: Do we need to add ".exe" in windows?
         name = raw_name.rsplit('.test', 1)[0]
         return not os.path.exists(os.path.join(path, name))
 
-    def _get_target_flags(self, cmake_vars):
-        build_type = cmake_vars["build_type"]
-        cflags = cmake_vars["c_flags"]
-        if build_type != "":
-            cflags = \
-                " ".join(cflags.split(" ") +
-                         cmake_vars["c_flags_"+build_type.lower()].split(" "))
-        return shlex.split(cflags)
-
-    def _get_cc_info(self):
+    def _extract_cmake_vars_from_cache(self):
         assert self.configured is True
         cmake_lah_output = self._check_output(
             [self.opts.cmake] + ['-LAH', '-N'] + [self._base_path])
         pattern2var = [
-            (re.compile("^%s:[^=]*=(.*)$" % cmakevar), var)
-            for cmakevar, var in (
-                ("CMAKE_C_COMPILER", "cc"),
-                ("CMAKE_BUILD_TYPE", "build_type"),
-                ("CMAKE_CXX_FLAGS", "cxx_flags"),
-                ("CMAKE_CXX_FLAGS_DEBUG", "cxx_flags_debug"),
-                ("CMAKE_CXX_FLAGS_MINSIZEREL", "cxx_flags_minsizerel"),
-                ("CMAKE_CXX_FLAGS_RELEASE", "cxx_flags_release"),
-                ("CMAKE_CXX_FLAGS_RELWITHDEBINFO", "cxx_flags_relwithdebinfo"),
-                ("CMAKE_C_FLAGS", "c_flags"),
-                ("CMAKE_C_FLAGS_DEBUG", "c_flags_debug"),
-                ("CMAKE_C_FLAGS_MINSIZEREL", "c_flags_minsizerel"),
-                ("CMAKE_C_FLAGS_RELEASE", "c_flags_release"),
-                ("CMAKE_C_FLAGS_RELWITHDEBINFO", "c_flags_relwithdebinfo"),)]
+            (re.compile("^%s:[^=]*=(.*)$" % cmakevar), cmakevar)
+            for cmakevar in (
+                "CMAKE_C_COMPILER",
+                "CMAKE_BUILD_TYPE",
+                "CMAKE_CXX_FLAGS",
+                "CMAKE_CXX_FLAGS_DEBUG",
+                "CMAKE_CXX_FLAGS_MINSIZEREL",
+                "CMAKE_CXX_FLAGS_RELEASE",
+                "CMAKE_CXX_FLAGS_RELWITHDEBINFO",
+                "CMAKE_C_FLAGS",
+                "CMAKE_C_FLAGS_DEBUG",
+                "CMAKE_C_FLAGS_MINSIZEREL",
+                "CMAKE_C_FLAGS_RELEASE",
+                "CMAKE_C_FLAGS_RELWITHDEBINFO",
+                "CMAKE_C_COMPILER_TARGET",
+                "CMAKE_CXX_COMPILER_TARGET",
+                )]
         cmake_vars = {}
         for line in cmake_lah_output.split("\n"):
             for pattern, varname in pattern2var:
                 m = re.match(pattern, line)
                 if m:
                     cmake_vars[varname] = m.group(1)
+        return cmake_vars
+
+    def _get_cc_info(self, cmake_vars):
+        build_type = cmake_vars["CMAKE_BUILD_TYPE"]
+        cflags = cmake_vars["CMAKE_C_FLAGS"]
+        if build_type != "":
+            cflags = \
+                " ".join(cflags.split(" ") +
+                         cmake_vars["CMAKE_C_FLAGS_" + build_type.upper()]
+                         .split(" "))
+        # FIXME: this probably needs to be conditionalized on the compiler
+        # being clang. Or maybe we need an
+        # lnt.testing.util.compilers.get_cc_info uses cmake somehow?
+        if "CMAKE_C_COMPILER_TARGET" in cmake_vars:
+            cflags += " --target=" + cmake_vars["CMAKE_C_COMPILER_TARGET"]
+        target_flags = shlex.split(cflags)
+
         return lnt.testing.util.compilers.get_cc_info(
-            cmake_vars["cc"], self._get_target_flags(cmake_vars))
+            cmake_vars["CMAKE_C_COMPILER"], target_flags)
 
-    def _parse_lit_output(self, path, data, only_test=False):
+    def _parse_lit_output(self, path, data, cmake_vars, only_test=False):
         LIT_METRIC_TO_LNT = {
             'compile_time': 'compile',
             'exec_time': 'exec',
             'score': 'score',
             'hash': 'hash',
             'link_time': 'compile',
             'size.__text': 'code_size'
         }
def _parse_lit_output(self, path, data, cmake_vars, only_test=False):
             # If we got this far, the result we were looking for didn't exist.
             raise RuntimeError("Result %s did not exist!" %
                                self.opts.single_result)
 
         # FIXME: Add more machine info!
         run_info = {
             'tag': 'nts'
         }
-        run_info.update(self._get_cc_info())
+        run_info.update(self._get_cc_info(cmake_vars))
         run_info['run_order'] = run_info['inferred_run_order']
         if self.opts.run_order:
             run_info['run_order'] = self.opts.run_order
 
         machine_info = {
         }
 
         machine = lnt.testing.Machine(self.nick, machine_info)

lnt/trunk/tests/SharedInputs/FakeCompilers/fakecompiler.py

class FakeCompiler(object):
     compiler_name = None
 
     def print_verbose_info(self):
         raise NotImplementedError
 
     def print_dumpmachine(self):
         raise NotImplementedError
 
-    def print_llvm_target(self):
+    def print_llvm_target(self, args):
         raise NotImplementedError
 
     def print_as_version(self):
         print >>sys.stderr, """(assembler version goes here)"""
 
     def print_ld_version(self):
         print >>sys.stderr, """(linker version goes here)"""
     
 class ICCv12_1_3(FakeCompiler):
     compiler_name = "icc-12.1.3"
     def print_verbose_info(self):
         print >>sys.stderr, """\
 icc: command line warning #10006: ignoring unknown option '-###'
 icc version 12.1.3 (gcc version 4.2.1 compatibility)
 /usr/bin/icc-2011-base/bin/intel64/mcpcom    -_g -mP3OPT_inline_alloca -D__HONOR_STD -D__ICC=1210 -D__INTEL_COMPILER=1210 "-_Acpu(x86_64)" "-_Amachine(x86_64)" -D__BLOCKS__ -D__PTRDIFF_TYPE__=long "-D__SIZE_TYPE__=unsigned long" -D__WCHAR_TYPE__=int -D__WINT_TYPE__=int "-D__INTMAX_TYPE__=long int" "-D__UINTMAX_TYPE__=long unsigned int" -D__LONG_MAX__=9223372036854775807L -D__QMSPP_ -D__OPTIMIZE__ -D__NO_MATH_INLINES -D__NO_STRING_INLINES -D__NO_INLINE__ -D__GNUC_GNU_INLINE__ -D__GNUC__=4 -D__GNUC_MINOR__=2 -D__GNUC_PATCHLEVEL__=1 -D__APPLE_CC__=5658 -D__ENVIRONMENT_MAC_OS_X_VERSION_MIN_REQUIRED__=1073 -D__LITTLE_ENDIAN__ -D__DYNAMIC__ "-D__private_extern__=__attribute__((visibility("hidden")))" -D__LP64__ -D_LP64 -D__GXX_ABI_VERSION=1002 -D__USER_LABEL_PREFIX__=_ -D__REGISTER_PREFIX__= -D__INTEL_RTTI__ -D__x86_64 -D__x86_64__ -D_MT -D__INTEL_COMPILER_BUILD_DATE=20120130 -D__PIC__ -D__APPLE__ -D__MACH__ -D__pentium4 -D__pentium4__ -D__tune_pentium4__ -D__SSE2__ -D__SSE3__ -D__SSSE3__ -D__SSE__ -D__MMX__ -_k -_8 -_l -_D -_a -_b -E --gnu_version=421 -_W5 --gcc-extern-inline --multibyte_chars --blocks --array_section --simd --simd_func -mP1OPT_print_version=FALSE -mP1OPT_version=12.1-intel64 -mGLOB_diag_use_message_catalog=FALSE /dev/null
 ... more boring stuff here ...
 """
 
-    def print_llvm_target(self):
+    def print_llvm_target(self, args):
         print """\
 icc: command line warning #10006: ignoring unknown option '-flto'
   .file "null"
   .section  __DATA, __data
 # End
   .subsections_via_symbols
 """
 
     def print_dumpmachine(self):
         print """i686-apple-darwin11"""
 
 class LLVMCompiler(FakeCompiler):
-    def print_llvm_target(self):
+    def print_llvm_target(self, args):
+        target = "x86_64-apple-darwin11.0.0"
+        for arg in args:
+            if arg.startswith("--target="):
+                target = arg[len("--target="):]
         print """\
 ; ModuleID = '/dev/null'
 target datalayout = "e-p:64:64:64-i1:8:8-i8:8:8-i16:16:16-i32:32:32-i64:64:64-\
 f32:32:32-f64:64:64-v64:64:64-v128:128:128-a0:0:64-s0:64:64-f80:128:128-\
 n8:16:32:64"
-target triple = "x86_64-apple-darwin11.0.0"
-"""
+target triple = "%s"
+""" % target
 
     def print_dumpmachine(self):
         print """x86_64-apple-darwin11.0.0"""
 
 # Clang build at r154331 (for example).
 class Clang_r154331(LLVMCompiler):
     compiler_name = "clang-r154331"
 
if '-dumpmachine' in args:
         compiler_instance.print_dumpmachine()
     elif args_contained_in(args, ('-v', '-###')):
         compiler_instance.print_verbose_info()
     elif 'Wa,-v' in args:
         compiler_instance.print_as_version()
     elif 'Wl,-v' in args:
         compiler_instance.print_ld_version()
     elif args_contained_in(args, ('-S', '-flto', '-o', '-', '/dev/null')):
-        compiler_instance.print_llvm_target()
+        compiler_instance.print_llvm_target(args)
     else:
         raise SystemExit("unrecognized argument vector: %r" % (
                 args,))
 
 if __name__ == '__main__':
     main()

lnt/trunk/tests/runtest/Inputs/test-suite-cmake/fake-cmake

 #!/bin/bash
 
 CMAKE_SRC_DIR="notfound"
 DUMP_VARS=false
 CMAKE_C_COMPILER=""
 CMAKE_CXX_COMPILER=""
+CMAKE_C_COMPILER_TARGET="notfound"
 
 while test $# -gt 0
 do
   if [[ $1 == "-C" ]]; then
     # If we passed a cache, just drop it and look like cmake.
     echo "Cmake Cache $2"
     shift
     shift
   elif [[ $1 == -D* ]]; then
     # handle -D arguments to cmake.
     if [[ $1 == -DCMAKE_C_COMPILER:* ]]; then CMAKE_C_COMPILER=${1#*=}; fi
     if [[ $1 == -DCMAKE_CXX_COMPILER:* ]]; then CMAKE_CXX_COMPILER=${1#*=}; fi
+    if [[ $1 == -DCMAKE_C_COMPILER_TARGET:* ]]; then CMAKE_C_COMPILER_TARGET=${1#*=}; fi
     shift
   elif [[ $1 == -LAH && $2 == -N ]]; then
     DUMP_VARS=true
     shift
     shift
   elif [[ ! -f $1/CMakeLists.txt && ! -f $1/CMakeCache.txt ]]; then
     # arguments not starting with -D or -C are assumed to point to the cmake
     # src or build dir
     exit 1
   else
     CMAKE_SRC_DIR=$1
     # if CMakeCache.txt exists, read in value for compiler
     if [[ -f $1/CMakeCache.txt ]]; then
         CMAKE_C_COMPILER=`grep CMAKE_C_COMPILER:FILEPATH= $1/CMakeCache.txt | cut -f2 -d'='`
         CMAKE_CXX_COMPILER=`grep CMAKE_CXX_COMPILER:FILEPATH= $1/CMakeCache.txt | cut -f2 -d'='`
+        CMAKE_C_COMPILER_TARGET=`grep CMAKE_C_COMPILER_TARGET:STRING= $1/CMakeCache.txt | cut -f2 -d'='`
     fi
     shift
   fi
 done
 
 if [[ $DUMP_VARS == "true" ]]
 then
   echo CMAKE_C_COMPILER:FILEPATH=$CMAKE_C_COMPILER
   echo CMAKE_CXX_COMPILER:FILEPATH=$CMAKE_CXX_COMPILER
+  if [[ $CMAKE_C_COMPILER_TARGET != "notfound" ]]; then
+    echo CMAKE_C_COMPILER_TARGET:STRING=$CMAKE_C_COMPILER_TARGET
+  fi
   echo CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
   echo CMAKE_C_FLAGS:STRING=-O0
   echo CMAKE_CXX_FLAGS:STRING=
   echo CMAKE_C_FLAGS_RELWITHDEBINFO:STRING=-O2 -g
   echo CMAKE_CXX_FLAGS_RELWITHDEBINFO:STRING=-O2 -g
 else
   cp $CMAKE_SRC_DIR/fake-test $CMAKE_SRC_DIR/fake-results.json $CMAKE_SRC_DIR/fake-results-profile.json .
   echo "Dummy" > CMakeCache.txt
   echo CMAKE_C_COMPILER:FILEPATH=$CMAKE_C_COMPILER >> CMakeCache.txt
   echo CMAKE_CXX_COMPILER:FILEPATH=$CMAKE_CXX_COMPILER >> CMakeCache.txt
+  if [[ $CMAKE_C_COMPILER_TARGET != "notfound" ]]; then
+    echo CMAKE_C_COMPILER_TARGET:STRING=$CMAKE_C_COMPILER_TARGET >> CMakeCache.txt
+  fi
   mkdir subtest
   cp $CMAKE_SRC_DIR/fake-test $CMAKE_SRC_DIR/fake-results.json subtest
 fi
 exit 0

lnt/trunk/tests/runtest/test_suite.py

 # RUN:     --sandbox %t.SANDBOX \
 # RUN:     --no-timestamp \
 # RUN:     --test-suite %S/Inputs/test-suite-cmake \
 # RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \
 # RUN:     --use-make %S/Inputs/test-suite-cmake/fake-make \
 # RUN:     --use-lit %S/Inputs/test-suite-cmake/fake-lit \
 # RUN:     > %t.log 2> %t.err || true
 # RUN: FileCheck  --check-prefix CHECK-MISSING-CC < %t.err %s
-# CHECK-MISSING-CC: error: --cc is required
+# CHECK-MISSING-CC: error: Couldn't find C compiler (). Maybe you should specify --cc?
 
 # Check on conflicting -cc and -cmake-define=CMAKE_C_COMPILER
 # options, the right compiler gets stored in the json report
 # RUN: lnt runtest test-suite \
 # RUN:     --sandbox %t.SANDBOX \
 # RUN:     --no-timestamp \
 # RUN:     --test-suite %S/Inputs/test-suite-cmake \
 # RUN:     --cmake-define=CMAKE_C_COMPILER:STRING=%{shared_inputs}/FakeCompilers/clang-r154332 \
 # RUN:     --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
 # RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \
 # RUN:     --use-make %S/Inputs/test-suite-cmake/fake-make \
 # RUN:     --use-lit %S/Inputs/test-suite-cmake/fake-lit \
 # RUN:     > %t.log 2> %t.err || true
 # RUN: FileCheck --check-prefix CHECK-CC-CONFL-CMAKEDEFINE < %t.SANDBOX/build/report.json %s
 # CHECK-CC-CONFL-CMAKEDEFINE: "run_order": "154332"
 
+# Check that while cross-compiling, the target architecture is recognized
+# correctly.
+# RUN: lnt runtest test-suite \
+# RUN:     --sandbox %t.SANDBOX \
+# RUN:     --no-timestamp \
+# RUN:     --test-suite %S/Inputs/test-suite-cmake \
+# RUN:     --cmake-define=CMAKE_C_COMPILER_TARGET:STRING=targetarch-linux-gnu \
+# RUN:     --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
+# RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \
+# RUN:     --use-make %S/Inputs/test-suite-cmake/fake-make \
+# RUN:     --use-lit %S/Inputs/test-suite-cmake/fake-lit \
+# RUN:     > %t.log 2> %t.err || true
+# RUN: FileCheck --check-prefix CHECK-CROSS-TARGET < %t.SANDBOX/build/report.json %s
+# CHECK-CROSS-TARGET: "cc_target": "targetarch-linux-gnu"
 
 # Check running with PGO
 # RUN: lnt runtest test-suite \
 # RUN:     --sandbox %t.SANDBOX \
 # RUN:     --no-timestamp \
 # RUN:     --test-suite %S/Inputs/test-suite-cmake \
 # RUN:     --cc %{shared_inputs}/FakeCompilers/clang-r154331 \
 # RUN:     --use-cmake %S/Inputs/test-suite-cmake/fake-cmake \