build.py: Implement "gcc" builder

[labath]

Summary

This implements the gcc builder in build.py script to allow it to
compile host executables when running on a non-windows host. Where it
made sense, I tried to share code with the msvc builder by moving stuff
to the base class.

I did not implement the lto part because I don't know how that works,
(and it's likely things will differ between platforms there).

[stella.stamenova]

A couple of small comments, but it looks good otherwise. Thanks!

lit/BuildScript/toolchain-clang.test

1 ↗

(On Diff #177195)

There are a couple of tests for the build script that we should enable on non-windows as well:

• script-args
• modes

Both should pass with your change

lit/helper/build.py

627 ↗

(On Diff #177195)

You can just add the thinlto command here. That should give you parity with the msvc builder:

args.append('-flto=thin')

[labath]

lit/BuildScript/toolchain-clang.test

1 ↗

(On Diff #177195)

I've enabled the two tests. I've had to tweak the "modes" test a bit to account for the different object file extension (.obj vs .o).

[zturner]

lldb/trunk/lit/helper/build.py

630

Why do we need this?

[labath]

lldb/trunk/lit/helper/build.py

630

Without this I got errors when trying this out on a mac (something about "dynamically linked executables must link to libSystem.dylib"). Doing a static link in this case did not seem like a bad choice, since this script doesn't even support building shared libraries at this point. If we get to a point where we want to build shared libraries in --nodefaultlib mode, then we will have to revisit this.

(this could be made darwin-only, but it seemed better to be consistent).

[stella.stamenova]

I am trying to use the new builder to build the lldb-mi tests, so that they can start consistently passing on Windows, so they're now using the gcc builder on Linux and failing:

2018-12-14T19:24:41.8890855Z ********************
2018-12-14T19:24:41.8904193Z Testing: 0 .. 10.. 20.. 30.
2018-12-14T19:24:41.8918035Z FAIL: LLDB :: tools/lldb-mi/exec/exec-interrupt.test (17141 of 45941)
2018-12-14T19:24:41.8932384Z ******************** TEST 'LLDB :: tools/lldb-mi/exec/exec-interrupt.test' FAILED ********************
2018-12-14T19:24:41.8945979Z Script:
2018-12-14T19:24:41.8959648Z --
2018-12-14T19:24:41.8974664Z : 'RUN: at line 1';   '/usr/bin/python2.7' /vstsdrive/_work/42/s/llvm/tools/lldb/lit/helper/build.py --compiler=any --arch=64 --tools-dir=/vstsdrive/_work/42/b/LLVMBuild/./bin /vstsdrive/_work/42/s/llvm/tools/lldb/lit/tools/lldb-mi/exec/inputs/main.c --nodefaultlib -o /vstsdrive/_work/42/b/LLVMBuild/tools/lldb/lit/tools/lldb-mi/exec/Output/exec-interrupt.test.tmp
2018-12-14T19:24:41.8991443Z : 'RUN: at line 2';   /vstsdrive/_work/42/b/LLVMBuild/bin/lldb-mi --synchronous /vstsdrive/_work/42/b/LLVMBuild/tools/lldb/lit/tools/lldb-mi/exec/Output/exec-interrupt.test.tmp < /vstsdrive/_work/42/s/llvm/tools/lldb/lit/tools/lldb-mi/exec/exec-interrupt.test | /vstsdrive/_work/42/b/LLVMBuild/bin/FileCheck /vstsdrive/_work/42/s/llvm/tools/lldb/lit/tools/lldb-mi/exec/exec-interrupt.test
2018-12-14T19:24:41.9006101Z --
2018-12-14T19:24:41.9019550Z Exit Code: 1
2018-12-14T19:24:41.9026154Z 
2018-12-14T19:24:41.9039902Z Command Output (stdout):
2018-12-14T19:24:41.9053450Z --
2018-12-14T19:24:41.9066966Z Cleaning exec-interrupt.test.tmp-main.o
2018-12-14T19:24:41.9081046Z Cleaning exec-interrupt.test.tmp
2018-12-14T19:24:41.9087711Z 
2018-12-14T19:24:41.9094525Z 
2018-12-14T19:24:41.9101310Z 
2018-12-14T19:24:41.9114993Z compiling main.c -> exec-interrupt.test.tmp-main.o
2018-12-14T19:24:41.9128817Z   STDOUT:
2018-12-14T19:24:41.9135428Z 
2018-12-14T19:24:41.9142241Z 
2018-12-14T19:24:41.9149031Z 
2018-12-14T19:24:41.9155781Z 
2018-12-14T19:24:41.9169689Z linking exec-interrupt.test.tmp-main.o -> exec-interrupt.test.tmp
2018-12-14T19:24:41.9183153Z   STDOUT:
2018-12-14T19:24:41.9190134Z 
2018-12-14T19:24:41.9203611Z   STDERR:
2018-12-14T19:24:41.9217412Z     clang-8: error: unable to execute command: Executable "ld" doesn't exist!
2018-12-14T19:24:41.9231640Z     clang-8: error: linker command failed with exit code 1 (use -v to see invocation)
2018-12-14T19:24:41.9238695Z 
2018-12-14T19:24:41.9245289Z 
2018-12-14T19:24:41.9258832Z --

lldb/trunk/lit/Breakpoint/case-sensitive.test

 # REQUIRES: nowindows
 #
-# RUN: %clang %p/Inputs/case-sensitive.c -g -o %t
+# RUN: %build %p/Inputs/case-sensitive.c --nodefaultlib -o %t
 # RUN: lldb-test breakpoints %t %s | FileCheck %s
 
 breakpoint set -f case-sensitive.c -l 3
 # CHECK-LABEL: breakpoint set -f case-sensitive.c -l 3
 # CHECK: At least one location.
 
 breakpoint set -f %p/Inputs/case-sensitive.c -l 3
 # CHECK-LABEL: breakpoint set -f {{.*}}/Inputs/case-sensitive.c -l 3

lldb/trunk/lit/BuildScript/modes.test

-REQUIRES: system-windows
-
 RUN: %build -n --verbose --arch=32 --mode=compile --compiler=any -o %t/foo.out foobar.c \
 RUN:    | FileCheck --check-prefix=COMPILE %s
 
 RUN: %build -n --verbose --arch=32 --mode=compile --compiler=any --outdir %t foo.c bar.c \
 RUN:    | FileCheck --check-prefix=COMPILE-MULTI %s
 
 RUN: %build -n --verbose --arch=32 --mode=link --compiler=any -o %t/foo.exe foobar.obj \
 RUN:    | FileCheck --check-prefix=LINK %s
 
 RUN: %build -n --verbose --arch=32 --mode=link --compiler=any -o %t/foobar.exe foo.obj bar.obj \
 RUN:    | FileCheck --check-prefix=LINK-MULTI %s
 
 RUN: %build -n --verbose --arch=32 --mode=compile-and-link --compiler=any -o %t/foobar.exe foobar.c \
 RUN:    | FileCheck --check-prefix=BOTH %s
 
 RUN: %build -n --verbose --arch=32 --mode=compile-and-link --compiler=any -o %t/foobar.exe foo.c bar.c \
 RUN:    | FileCheck --check-prefix=BOTH-MULTI %s
 
 
 COMPILE: compiling foobar.c -> foo.out
 
-COMPILE-MULTI: compiling foo.c -> foo.obj
-COMPILE-MULTI: compiling bar.c -> bar.obj
+COMPILE-MULTI: compiling foo.c -> foo.o{{(bj)?}}
+COMPILE-MULTI: compiling bar.c -> bar.o{{(bj)?}}
 
 
 LINK: linking foobar.obj -> foo.exe
 
 LINK-MULTI: linking foo.obj+bar.obj -> foobar.exe
 
-BOTH: compiling foobar.c -> foobar.exe-foobar.obj
-BOTH: linking foobar.exe-foobar.obj -> foobar.exe
+BOTH: compiling foobar.c -> [[OBJFOO:foobar.exe-foobar.o(bj)?]]
+BOTH: linking [[OBJFOO]] -> foobar.exe
 
-BOTH-MULTI: compiling foo.c -> foobar.exe-foo.obj
-BOTH-MULTI: compiling bar.c -> foobar.exe-bar.obj
-BOTH-MULTI: linking foobar.exe-foo.obj+foobar.exe-bar.obj -> foobar.exe
+BOTH-MULTI: compiling foo.c -> [[OBJFOO:foobar.exe-foo.o(bj)?]]
+BOTH-MULTI: compiling bar.c -> [[OBJBAR:foobar.exe-bar.o(bj)?]]
+BOTH-MULTI: linking [[OBJFOO]]+[[OBJBAR]] -> foobar.exe

lldb/trunk/lit/BuildScript/script-args.test

-REQUIRES: system-windows
-
 RUN: %build -n --verbose --arch=32 --mode=compile --compiler=any -o %t/foo.out foobar.c \
 RUN:    | FileCheck %s
 RUN: %build -n --verbose --arch=32 --mode=compile --compiler=any --outdir %t foo.c bar.c \
 RUN:    | FileCheck --check-prefix=MULTI-INPUT %s
 
 
 CHECK:      Script Arguments:
 CHECK-NEXT:   Arch: 32

 MULTI-INPUT-NEXT:   Output: 
 MULTI-INPUT-NEXT:   Nodefaultlib: False
 MULTI-INPUT-NEXT:   Opt: none
 MULTI-INPUT-NEXT:   Mode: compile
 MULTI-INPUT-NEXT:   Clean: True
 MULTI-INPUT-NEXT:   Verbose: True
 MULTI-INPUT-NEXT:   Dryrun: True
 MULTI-INPUT-NEXT:   Inputs: foo.c
-MULTI-INPUT-NEXT:           bar.c
+MULTI-INPUT-NEXT:           bar.c
 

lldb/trunk/lit/BuildScript/toolchain-clang.test

+RUN: %build -n --verbose --arch=32 --compiler=clang --mode=compile-and-link -o %t/foo.exe foobar.c \
+RUN:    | FileCheck --check-prefix=CHECK --check-prefix=CHECK-32 %s
+
+RUN: %build -n --verbose --arch=64 --compiler=clang --mode=compile-and-link -o %t/foo.exe foobar.c \
+RUN:    | FileCheck --check-prefix=CHECK --check-prefix=CHECK-64 %s
+
+CHECK: Cleaning {{.*}}toolchain-clang.test.tmp{{.}}foo.exe-foobar.o
+CHECK: Cleaning {{.*}}toolchain-clang.test.tmp{{.}}foo.exe
+CHECK: compiling foobar.c -> foo.exe-foobar.o
+CHECK-32: {{.*}}clang++{{(.exe)?}} -m32 -g -O0 -c -o {{.*}}foo.exe-foobar.o {{.*}}foobar.c
+CHECK-64: {{.*}}clang++{{(.exe)?}} -m64 -g -O0 -c -o {{.*}}foo.exe-foobar.o {{.*}}foobar.c
+CHECK: linking foo.exe-foobar.o -> foo.exe
+CHECK-32: {{.*}}clang++{{(.exe)?}} -m32 -o {{.*}}foo.exe {{.*}}foo.exe-foobar.o
+CHECK-64: {{.*}}clang++{{(.exe)?}} -m64 -o {{.*}}foo.exe {{.*}}foo.exe-foobar.o

lldb/trunk/lit/helper/build.py

+#! /usr/bin/env python
+
 from __future__ import print_function
 
 import argparse
 import os
 import signal
 import subprocess
 import sys
 
if name.startswith('clang'):
         return 'clang'
     if name.startswith('gcc') or name.startswith('g++'):
         return 'gcc'
     if name == 'cc' or name == 'c++':
         return 'generic'
     return 'unknown'
 
 class Builder(object):
-    def __init__(self, toolchain_type, args):
+    def __init__(self, toolchain_type, args, obj_ext):
         self.toolchain_type = toolchain_type
         self.inputs = args.inputs
         self.arch = args.arch
         self.opt = args.opt
         self.outdir = args.outdir
         self.compiler = args.compiler
         self.clean = args.clean
         self.output = args.output
         self.mode = args.mode
         self.nodefaultlib = args.nodefaultlib
         self.verbose = args.verbose
+        self.obj_ext = obj_ext
+
+    def _exe_file_name(self):
+        assert self.mode != 'compile'
+        return self.output
+
+    def _output_name(self, input, extension, with_executable=False):
+        basename = os.path.splitext(os.path.basename(input))[0] + extension
+        if with_executable:
+            exe_basename = os.path.basename(self._exe_file_name())
+            basename = exe_basename + '-' + basename
+
+        output = os.path.join(self.outdir, basename)
+        return os.path.normpath(output)
+
+    def _obj_file_names(self):
+        if self.mode == 'link':
+            return self.inputs
+
+        if self.mode == 'compile-and-link':
+            # Object file names should factor in both the input file (source)
+            # name and output file (executable) name, to ensure that two tests
+            # which share a common source file don't race to write the same
+            # object file.
+            return [self._output_name(x, self.obj_ext, True) for x in self.inputs]
+
+        if self.mode == 'compile' and self.output:
+            return [self.output]
+
+        return [self._output_name(x, self.obj_ext) for x in self.inputs]
+
+    def build_commands(self):
+        commands = []
+        if self.mode == 'compile' or self.mode == 'compile-and-link':
+            for input, output in zip(self.inputs, self._obj_file_names()):
+                commands.append(self._get_compilation_command(input, output))
+        if self.mode == 'link' or self.mode == 'compile-and-link':
+            commands.append(self._get_link_command())
+        return commands
+
 
 class MsvcBuilder(Builder):
     def __init__(self, toolchain_type, args):
-        Builder.__init__(self, toolchain_type, args)
+        Builder.__init__(self, toolchain_type, args, '.obj')
 
         self.msvc_arch_str = 'x86' if self.arch == '32' else 'x64'
 
         if toolchain_type == 'msvc':
             # Make sure we're using the appropriate toolchain for the desired
             # target type.
             compiler_parent_dir = os.path.dirname(self.compiler)
             selected_target_version = os.path.basename(compiler_parent_dir)
def _get_visual_studio_environment(self):
             compileenv['INCLUDE'] = os.pathsep.join(includes)
             compileenv.update(defaultenv)
         if libs:
             linkenv = {}
             linkenv['LIB'] = os.pathsep.join(libs)
             linkenv.update(defaultenv)
         return (compileenv, linkenv)
 
-    def _output_name(self, input, extension, with_executable=False):
-        basename = os.path.splitext(os.path.basename(input))[0] + extension
-        if with_executable:
-            exe_basename = os.path.basename(self._exe_file_name())
-            basename = exe_basename + '-' + basename
-
-        output = os.path.join(self.outdir, basename)
-        return os.path.normpath(output)
-
     def _ilk_file_names(self):
         if self.mode == 'link':
             return []
 
         return [self._output_name(x, '.ilk') for x in self.inputs]
 
-    def _obj_file_names(self):
-        if self.mode == 'link':
-            return self.inputs
-
-        if self.mode == 'compile-and-link':
-            # Object file names should factor in both the input file (source)
-            # name and output file (executable) name, to ensure that two tests
-            # which share a common source file don't race to write the same
-            # object file.
-            return [self._output_name(x, '.obj', True) for x in self.inputs]
-
-        if self.mode == 'compile' and self.output:
-            return [self.output]
-
-        return [self._output_name(x, '.obj') for x in self.inputs]
-
     def _pdb_file_name(self):
         if self.mode == 'compile':
             return None
         return os.path.splitext(self.output)[0] + '.pdb'
 
-    def _exe_file_name(self):
-        if self.mode == 'compile':
-            return None
-        return self.output
-
     def _get_compilation_command(self, source, obj):
         args = []
 
         args.append(self.compiler)
         if self.toolchain_type == 'clang-cl':
             args.append('-m' + self.arch)
 
         if self.opt == 'none':
def build_commands(self):
         if self.mode == 'compile' or self.mode == 'compile-and-link':
             for input, output in zip(self.inputs, self._obj_file_names()):
                 commands.append(self._get_compilation_command(input, output))
         if self.mode == 'link' or self.mode == 'compile-and-link':
             commands.append(self._get_link_command())
         return commands
 
     def output_files(self):
-        outdir = os.path.dirname(self.output)
-        file = os.path.basename(self.output)
-        name, ext = os.path.splitext(file)
-
         outputs = []
         if self.mode == 'compile' or self.mode == 'compile-and-link':
             outputs.extend(self._ilk_file_names())
             outputs.extend(self._obj_file_names())
         if self.mode == 'link' or self.mode == 'compile-and-link':
             outputs.append(self._pdb_file_name())
             outputs.append(self._exe_file_name())
 
         return [x for x in outputs if x is not None]
 
 class GccBuilder(Builder):
     def __init__(self, toolchain_type, args):
-        Builder.__init__(self, toolchain_type, args)
+        Builder.__init__(self, toolchain_type, args, '.o')
+
+    def _get_compilation_command(self, source, obj):
+        args = []
+
+        args.append(self.compiler)
+        args.append('-m' + self.arch)
+
+        args.append('-g')
+        if self.opt == 'none':
+            args.append('-O0')
+        elif self.opt == 'basic':
+            args.append('-O2')
+        elif self.opt == 'lto':
+            args.append('-flto=thin')
+        if self.nodefaultlib:
+            args.append('-nostdinc')
+            args.append('-static')
+        args.append('-c')
+
+        args.extend(['-o', obj])
+        args.append(source)
+
+        return ('compiling', [source], obj, {}, args)
+
+    def _get_link_command(self):
+        args = []
+        args.append(self.compiler)
+        args.append('-m' + self.arch)
+        if self.nodefaultlib:
+            args.append('-nostdlib')
+            args.append('-static')
+            main_symbol = 'main'
+            if sys.platform == 'darwin':
+                main_symbol = '_main'
+            args.append('-Wl,-e,' + main_symbol)
+        args.extend(['-o', self._exe_file_name()])
+        args.extend(self._obj_file_names())
+
+        return ('linking', self._obj_file_names(), self._exe_file_name(), {}, args)
 
-    def build_commands(self):
-        pass
 
     def output_files(self):
-        pass
+        outputs = []
+        if self.mode == 'compile' or self.mode == 'compile-and-link':
+            outputs.extend(self._obj_file_names())
+        if self.mode == 'link' or self.mode == 'compile-and-link':
+            outputs.append(self._exe_file_name())
+
+        return outputs
 
 def indent(text, spaces):
     def prefixed_lines():
         prefix = ' ' * spaces
         for line in text.splitlines(True):
             yield prefix + line
     return ''.join(prefixed_lines())