[lldb] Fix running tests when LLVM target triple is different from host

[alvinhochun]

Summary

This passes the host triple to the lldbsuite test builder used in API
tests and the build helper used in Shell tests, so that Clang can use
the correct host triple when LLVM was configured with a default target
triple different from the host triple.

[JDevlieghere]

Supporting builds with different host and target triples makes sense. However, I have a few concerns about the current patch:

• I wouldn't cal it the "host triple". The API tests (dotest.py) support running the test suite for a different platform/architecture. Take a look at the Darwin builder and you'll see a bunch of support for that.
• How does this interact with the -arch flag? When the host and target triple differ, which part of the triple is different? In theory it can be any part, but maybe your use case could be covered by setting the --arch flag?
• This doesn't update getTriple in builder.py which is necessary for some of the decorators to work (in particular the ones that invoke the compiler to see if something is supported).

[alvinhochun]

Thanks for the review.

In D134751#3818743, @JDevlieghere wrote:

Supporting builds with different host and target triples makes sense. However, I have a few concerns about the current patch:

• I wouldn't cal it the "host triple". The API tests (dotest.py) support running the test suite for a different platform/architecture. Take a look at the Darwin builder and you'll see a bunch of support for that.

How about calling it "clang triple"?

• How does this interact with the -arch flag? When the host and target triple differ, which part of the triple is different? In theory it can be any part, but maybe your use case could be covered by setting the --arch flag?

If I run Clang with --target=x86_64-linux-gnu -m32 then it automatically targets i386-linux-gnu, so I suppose -arch should override the arch part of the target, which is correct. DarwinBuilder::getArchCFlagson the other hand has a different handling which generates a -target option from arch and other Apple-specific details without considering the new host-triple flag (or whatever we end up calling it), so it should behave the same as before this change.

For me, my Linux native build has the default target triple configured to x86_64-w64-windows-gnu, and linking of lldb tests fail due to missing mingw-w64 libraries. (Even if they do link the tests probably won't work anyway.) The Shell tests that use the %clang_host lit substitution work because they include the host --target flag, so I thought the tests using the build scripts should do the same.

• This doesn't update getTriple in builder.py which is necessary for some of the decorators to work (in particular the ones that invoke the compiler to see if something is supported).

I see, I will change this too.

lldb/packages/Python/lldbsuite/test/builders/builder.py

def getExtraMakeArgs(self):
         """
         Helper function to return extra argumentsfor the make system. This
         method is meant to be overridden by platform specific builders.
         """
         return []
 
     def getArchCFlags(self, architecture):
         """Returns the ARCH_CFLAGS for the make system."""
+        if configuration.host_triple:
+            return ["ARCH_CFLAGS=--target=" + configuration.host_triple]
+        else:
             return []
 
     def getMake(self, test_subdir, test_name):
         """Returns the invocation for GNU make.
         The first argument is a tuple of the relative path to the testcase
         and its filename stem."""
         if platform.system() == "FreeBSD" or platform.system() == "NetBSD":
             make = "gmake"
         else:

lldb/packages/Python/lldbsuite/test/configuration.py

 lldb_framework_path = None
 
 # Test suite repeat count.  Can be overwritten with '-# count'.
 count = 1
 
 # The 'arch' and 'compiler' can be specified via command line.
 arch = None
 compiler = None
+host_triple = None
 dsymutil = None
 sdkroot = None
 
 # The overriden dwarf verison.
 dwarf_version = 0
 
 # Any overridden settings.
 settings = []

lldb/packages/Python/lldbsuite/test/dotest.py

else:
         else:
             # 'clang' on ubuntu 14.04 is 3.4 so we try clang-3.5 first
             candidateCompilers = ['clang-3.5', 'clang', 'gcc']
             for candidate in candidateCompilers:
                 if which(candidate):
                     configuration.compiler = candidate
                     break
 
+    configuration.host_triple = args.host_triple
+
     if args.dsymutil:
         configuration.dsymutil = args.dsymutil
     elif platform_system == 'Darwin':
         configuration.dsymutil = seven.get_command_output(
             'xcrun -find -toolchain default dsymutil')
     if args.llvm_tools_dir:
         configuration.filecheck = shutil.which("FileCheck", path=args.llvm_tools_dir)
         configuration.yaml2obj = shutil.which("yaml2obj", path=args.llvm_tools_dir)

lldb/packages/Python/lldbsuite/test/dotest_args.py

def create_parser():
     group.add_argument(
         '-A',
         '--arch',
         metavar='arch',
         dest='arch',
         help=textwrap.dedent('''Specify the architecture(s) to test. This option can be specified more than once'''))
     group.add_argument('-C', '--compiler', metavar='compiler', dest='compiler', help=textwrap.dedent(
         '''Specify the compiler(s) used to build the inferior executables. The compiler path can be an executable basename or a full path to a compiler executable. This option can be specified multiple times.'''))
+    group.add_argument('--host-triple', metavar='host_triple', dest='host_triple', help=textwrap.dedent(
+        '''Specify the host target triple used to build the inferior executables.'''))
     if sys.platform == 'darwin':
         group.add_argument('--apple-sdk', metavar='apple_sdk', dest='apple_sdk', default="", help=textwrap.dedent(
             '''Specify the name of the Apple SDK (macosx, macosx.internal, iphoneos, iphoneos.internal, or path to SDK) and use the appropriate tools from that SDK's toolchain.'''))
     group.add_argument('--libcxx-include-dir', help=textwrap.dedent('Specify the path to a custom libc++ include directory. Must be used in conjunction with --libcxx-library-dir.'))
     group.add_argument('--libcxx-library-dir', help=textwrap.dedent('Specify the path to a custom libc++ library directory. Must be used in conjunction with --libcxx-include-dir.'))
     # FIXME? This won't work for different extra flags according to each arch.
     group.add_argument(
         '-E',

lldb/test/API/lit.cfg.py

if is_configured('clang_module_cache'):
   dotest_cmd += ['--clang-module-cache-dir', config.clang_module_cache]
 
 if is_configured('lldb_executable'):
   dotest_cmd += ['--executable', config.lldb_executable]
 
 if is_configured('test_compiler'):
   dotest_cmd += ['--compiler', config.test_compiler]
 
+if config.host_triple != config.target_triple:
+  dotest_cmd += ['--host-triple', config.host_triple]
+
 if is_configured('dsymutil'):
   dotest_cmd += ['--dsymutil', config.dsymutil]
 
 if is_configured('llvm_tools_dir'):
   dotest_cmd += ['--llvm-tools-dir', config.llvm_tools_dir]
 
 if is_configured('server'):
   dotest_cmd += ['--server', config.server]

lldb/test/Shell/BuildScript/script-args.test

 RUN: %build -n --verbose --arch=32 --mode=compile --compiler=any -o %t/foo.out foobar.c \
 RUN:    | FileCheck %s
 RUN: %build -n --verbose --arch=32 --mode=compile --compiler=any --outdir %t foo.c bar.c \
 RUN:    | FileCheck --check-prefix=MULTI-INPUT %s
 
 
 CHECK:      Script Arguments:
 CHECK-NEXT:   Arch: 32
 CHECK:        Compiler: any
+CHECK:        Host Triple:
 CHECK:        Outdir: {{.*}}script-args.test.tmp
 CHECK:        Output: {{.*}}script-args.test.tmp{{.}}foo.out
 CHECK:        Nodefaultlib: False
 CHECK:        Opt: none
 CHECK:        Mode: compile
 CHECK:        Clean: True
 CHECK:        Verbose: True
 CHECK:        Dryrun: True
 CHECK:        Inputs: foobar.c
 
 MULTI-INPUT:      Script Arguments:
 MULTI-INPUT-NEXT:   Arch: 32
 MULTI-INPUT-NEXT:   Compiler: any
+MULTI-INPUT-NEXT:   Host Triple:
 MULTI-INPUT-NEXT:   Outdir: {{.*}}script-args.test.tmp
 MULTI-INPUT-NEXT:   Output: 
 MULTI-INPUT-NEXT:   Nodefaultlib: False
 MULTI-INPUT-NEXT:   Opt: none
 MULTI-INPUT-NEXT:   Mode: compile
 MULTI-INPUT-NEXT:   Clean: True
 MULTI-INPUT-NEXT:   Verbose: True
 MULTI-INPUT-NEXT:   Dryrun: True
 MULTI-INPUT-NEXT:   Inputs: foo.c
 MULTI-INPUT-NEXT:           bar.c

lldb/test/Shell/BuildScript/toolchain-clang.test

-RUN: %build -n --verbose --arch=32 --compiler=clang --mode=compile-and-link -o %t/foo.exe foobar.c \
+RUN: %build -n --verbose --arch=32 --compiler=clang --host-triple= --mode=compile-and-link -o %t/foo.exe foobar.c \
 RUN:    | FileCheck --check-prefix=CHECK --check-prefix=CHECK-32 %s
 
-RUN: %build -n --verbose --arch=64 --compiler=clang --mode=compile-and-link -o %t/foo.exe foobar.c \
+RUN: %build -n --verbose --arch=64 --compiler=clang --host-triple= --mode=compile-and-link -o %t/foo.exe foobar.c \
 RUN:    | FileCheck --check-prefix=CHECK --check-prefix=CHECK-64 %s
 
+RUN: %build -n --verbose --arch=64 --compiler=clang --host-triple=x86_64-linux-gnu --mode=compile-and-link -o %t/foo.exe foobar.c \
+RUN:    | FileCheck --check-prefix=CHECK --check-prefix=CHECK-TARGET %s
+
 CHECK: Cleaning {{.*}}toolchain-clang.test.tmp{{.}}foo.exe-foobar.o
 CHECK: Cleaning {{.*}}toolchain-clang.test.tmp{{.}}foo.exe
 CHECK: compiling foobar.c -> foo.exe-foobar.o
 CHECK-32: {{.*}}clang++{{(\.EXE)?}} -m32 -g -O0 -c -o {{.*}}foo.exe-foobar.o {{.*}}foobar.c
 CHECK-64: {{.*}}clang++{{(\.EXE)?}} -m64 -g -O0 -c -o {{.*}}foo.exe-foobar.o {{.*}}foobar.c
+CHECK-TARGET: {{.*}}clang++{{(\.EXE)?}} --target=x86_64-linux-gnu -m64 -g -O0 -c -o {{.*}}foo.exe-foobar.o {{.*}}foobar.c
 CHECK: linking foo.exe-foobar.o -> foo.exe
 CHECK-32: {{.*}}clang++{{(\.EXE)?}} -m32 {{(-L.* )?(-Wl,-rpath,.* )?}}-o {{.*}}foo.exe {{.*}}foo.exe-foobar.o
 CHECK-64: {{.*}}clang++{{(\.EXE)?}} -m64 {{(-L.* )?(-Wl,-rpath,.* )?}}-o {{.*}}foo.exe {{.*}}foo.exe-foobar.o
+CHECK-TARGET: {{.*}}clang++{{(\.EXE)?}} --target=x86_64-linux-gnu -m64 {{(-L.* )?(-Wl,-rpath,.* )?}}-o {{.*}}foo.exe {{.*}}foo.exe-foobar.o

lldb/test/Shell/helper/build.py

parser.add_argument('--arch',
                     help='Specify the architecture to target.')
 
 parser.add_argument('--compiler',
                     metavar='compiler',
                     dest='compiler',
                     required=True,
                     help='Path to a compiler executable, or one of the values [any, msvc, clang-cl, gcc, clang]')
 
+parser.add_argument('--host-triple',
+                    metavar='host_triple',
+                    dest='host_triple',
+                    required=False,
+                    help='The host target triple to use instead of the default')
+
 parser.add_argument('--libs-dir',
                     metavar='directory',
                     dest='libs_dir',
                     required=False,
                     action='append',
                     help='If specified, a path to linked libraries to be passed via -L')
 
 parser.add_argument('--tools-dir',

 class Builder(object):
     def __init__(self, toolchain_type, args, obj_ext):
         self.toolchain_type = toolchain_type
         self.inputs = args.inputs
         self.arch = args.arch
         self.opt = args.opt
         self.outdir = args.outdir
         self.compiler = args.compiler
+        self.host_triple = args.host_triple
         self.clean = args.clean
         self.output = args.output
         self.mode = args.mode
         self.nodefaultlib = args.nodefaultlib
         self.verbose = args.verbose
         self.obj_ext = obj_ext
         self.lib_paths = args.libs_dir
 
def __init__(self, toolchain_type, args):
         if sys.platform == 'darwin':
             cmd = ['xcrun', '--sdk', args.apple_sdk, '--show-sdk-path']
             self.apple_sdk = subprocess.check_output(cmd).strip().decode('utf-8')
 
     def _get_compilation_command(self, source, obj):
         args = []
 
         args.append(self.compiler)
+        if self.host_triple:
+            args.append('--target=' + self.host_triple)
         args.append('-m' + self.arch)
 
         args.append('-g')
         if self.opt == 'none':
             args.append('-O0')
         elif self.opt == 'basic':
             args.append('-O2')
         elif self.opt == 'lto':
def _get_compilation_command(self, source, obj):
         if sys.platform == 'darwin':
             args.extend(['-isysroot', self.apple_sdk])
 
         return ('compiling', [source], obj, None, args)
 
     def _get_link_command(self):
         args = []
         args.append(self.compiler)
+        if self.host_triple:
+            args.append('--target=' + self.host_triple)
         args.append('-m' + self.arch)
         if self.nodefaultlib:
             args.append('-nostdlib')
             args.append('-static')
             main_symbol = 'main'
             if sys.platform == 'darwin':
                 main_symbol = '_main'
             args.append('-Wl,-e,' + main_symbol)

 if not toolchain_path or not toolchain_type:
     print('Unable to find toolchain {0}'.format(args.compiler))
     sys.exit(1)
 
 if args.verbose:
     print('Script Arguments:')
     print('  Arch: ' + args.arch)
     print('  Compiler: ' + args.compiler)
+    print('  Host Triple: ' + (args.host_triple or ""))
     print('  Outdir: ' + args.outdir)
     print('  Output: ' + args.output)
     print('  Nodefaultlib: ' + str(args.nodefaultlib))
     print('  Opt: ' + args.opt)
     print('  Mode: ' + args.mode)
     print('  Clean: ' + str(args.clean))
     print('  Verbose: ' + str(args.verbose))
     print('  Dryrun: ' + str(args.dry))

lldb/test/Shell/helper/toolchain.py

def use_lldb_substitutions(config):
     dsname = 'debugserver' if platform.system() in ['Darwin'] else 'lldb-server'
     dsargs = [] if platform.system() in ['Darwin'] else ['gdbserver']
 
     build_script = os.path.dirname(__file__)
     build_script = os.path.join(build_script, 'build.py')
     build_script_args = [build_script,
                         '--compiler=any', # Default to best compiler
                         '--arch=' + str(config.lldb_bitness)]
+    if config.host_triple != config.target_triple:
+        build_script_args += ['--host-triple', config.host_triple]
     if config.lldb_lit_tools_dir:
         build_script_args.append('--tools-dir={0}'.format(config.lldb_lit_tools_dir))
     if config.lldb_tools_dir:
         build_script_args.append('--tools-dir={0}'.format(config.lldb_tools_dir))
     if config.llvm_libs_dir:
         build_script_args.append('--libs-dir={0}'.format(config.llvm_libs_dir))
 
     lldb_init = _get_lldb_init_path(config)