Introduce Fuchsia builder

[phosek]

Summary

This change introduces Fuchsia builder. This builder compiles a
complete Fuchsia toolchain in Linux host, using a two-stage build.

[gkistanova]

Hello Petr,

Thanks for working on this!

On what commits do you want this builder to do a build?
As it is currently coded it would never work.

Please also see the in-lined notes.

zorg/buildbot/builders/FuchsiaBuilder.py

19 ↗

(On Diff #179027)

Would you not use mutable default arguments in function definitions,please? This has quite bad side effects which are hard to debug. You can find more details at https://docs.quantifiedcode.com/python-anti-patterns/correctness/mutable_default_value_as_argument.html.

20 ↗

(On Diff #179027)

The same is here.

36 ↗

(On Diff #179027)

You do not need the get_builddir property, really. The current build directory is always available as a build property and could be accessed as %(workdir)s everywhere below instead of %(builddir)s .

83 ↗

(On Diff #179027)

Could you also support the clean and clean_obj build properties, please? To have this unified with other builders.

86 ↗

(On Diff #179027)

Could you flip the condition over and check it for is None, instead, please? This is a cosmetic, and would help reading the code.

[phosek]

In D55917#1344065, @gkistanova wrote:

On what commits do you want this builder to do a build?
As it is currently coded it would never work.

Thanks for the review! I noticed the missing scheduler after you pointed it out and I've added a new scheduler for this builder.

Do you know if there is there a way to test the builder factory manually?

[gkistanova]

It is better to use the zorg.buildbot.process.factory.LLVMBuildFactory instead of buildbot.process.factory.BuildFactory(). Zorg would assign it to the correct scheduler automatically in this case.

Anyway, there is still a few problems with scheduling this builder.

The scheduler listens for commits in SVN as the only point if truth. But you are checking out the source code from github. There is a racing condition, as the change might not yet be ready in the github repo. And revision number provided by the scheduler would not work for github as expected either.

May I suggest checking out from SVN till we would officially move to github?

[phosek]

In D55917#1345911, @gkistanova wrote:

It is better to use the zorg.buildbot.process.factory.LLVMBuildFactory instead of buildbot.process.factory.BuildFactory(). Zorg would assign it to the correct scheduler automatically in this case.

I'm fine switching to LLVMBuildFactory to make it more unified with other builders.

Anyway, there is still a few problems with scheduling this builder.

The scheduler listens for commits in SVN as the only point if truth. But you are checking out the source code from github. There is a racing condition, as the change might not yet be ready in the github repo. And revision number provided by the scheduler would not work for github as expected either.

May I suggest checking out from SVN till we would officially move to github?

I could do that, but our builder already assumes the flat (monorepo) layout so I'd need to implement the SVN checkout separately rather than relying on LLVMBuildFactory which uses the hierarchical layout.

During the GitHub migration panel at the last Developers' Meeting, it was suggested that after the new repository is available which according to last email on the thread should be some time next week, bot maintainers should start migrating away from SVN to Git. Do you have any estimate for when the LLVMPoller will be ready to support the new Git repository as well? I'd be fine waiting a few more days and start immediately on Git, but if it's going to take longer, I'd be fine starting with SVN and then switch to Git later.

[gkistanova]

Thanks,

Could you clean off the unused imports, please?
You could test how the factory works on the silent master. I'll help if any debugging would be needed.

[phosek]

In D55917#1348700, @gkistanova wrote:

Could you clean off the unused imports, please?

Done

[gkistanova]

Thanks, Petr!
LGTM

zorg/trunk/buildbot/osuosl/master/config/builders.py

 from zorg.buildbot.builders import ABITestsuitBuilder
 from zorg.buildbot.builders import ClangLTOBuilder3Stage
 from zorg.buildbot.builders import ClangLTOBuilder
 from zorg.buildbot.builders import UnifiedTreeBuilder
 from zorg.buildbot.builders import CUDATestsuiteBuilder
 from zorg.buildbot.builders import AOSPBuilder
 from zorg.buildbot.builders import AnnotatedBuilder
 from zorg.buildbot.builders import LLDPerformanceTestsuite
+from zorg.buildbot.builders import FuchsiaBuilder
 
 # Plain LLVM builders.
 def _get_llvm_builders():
     return [
         # We currently have to force LLVM_HOST_TRIPLE and
         # LLVM_DEFAULT_TARGET_TRIPLE on this system. CMake gets the value
         # correct for the processor but it's currently not possible to emit O32
         # code using a mips64-* triple. This is a bug and should be fixed soon.
return [
             cmake_extra_opts={'LIBCXXABI_USE_LLVM_UNWINDER': 'ON',
                               'LIBCXX_ENABLE_EXCEPTIONS': 'OFF',
                               'LIBCXXABI_ENABLE_EXCEPTIONS': 'OFF',
                               'CMAKE_C_FLAGS': '-mcpu=cortex-a57',
                               'CMAKE_CXX_FLAGS': '-mcpu=cortex-a57'})},
 
     ]
 
+# Toolchain builders.
+def _get_toolchain_builders():
+      return [
+          {'name': "fuchsia-x86_64-linux",
+           'slavenames' :["fuchsia-debian-64-us-central1-a-1", "fuchsia-debian-64-us-central1-b-1"],
+           'builddir': "fuchsia-x86_64-linux",
+           'factory': FuchsiaBuilder.getToolchainBuildFactory()
+           'category': 'fuchsia'},
+        ]
+
 # Experimental and stopped builders
 def _get_on_demand_builders():
     return [
         ]
 
 def _get_experimental_scheduled_builders():
     return [
         {'name' : "clang-bpf-build",
def get_builders():
     for b in _get_openmp_builders():
         b['category'] = 'openmp'
         yield b
 
     for b in _get_libcxx_builders():
         b['category'] = 'libcxx'
         yield b
 
+    for b in _get_toolchain_builders():
+        b['category'] = 'toolchain'
+        yield b
+
     for b in _get_documentation_builders():
         yield b
 
     for b in _get_experimental_scheduled_builders():
         if not b.get('category', '').endswith('.exp'):
            b['category'] = b.get('category', '') + '.exp'
         yield b
 
     for b in _get_on_demand_builders():
         if not b.get('category', '').endswith('.on-demand'):
            b['category'] = b.get('category', '') + '.on-demand'
         yield b

zorg/trunk/buildbot/osuosl/master/config/slaves.py

return [
         create_slave("avx2-intel64", properties={'jobs': 80}, max_builds=1),
 
         # X86_64 with SDE, Ubuntu 16.04.2, Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
         create_slave("sde-avx512-intel64", properties={'jobs': 80}, max_builds=1),
 
         # FreeBSD 11 amd64
         create_slave("freebsd11-amd64", properties={'jobs': 2}, max_builds=1),
 
+        # Debian 9.0 x86_64 64-core GCE instances
+        create_slave("fuchsia-debian-64-us-central1-a-1", properties={'jobs': 64}, max_builds=1),
+        create_slave("fuchsia-debian-64-us-central1-b-1", properties={'jobs': 64}, max_builds=1),
+
         # Defunct.
         # Debian 86_64, 2 x 6-core Opteron 2.6 GHz
         #create_slave("osu8", properties={'jobs' : 6}, max_builds=2),
 
         # OpenBSD amd64
         create_slave("openbsd-amd64", properties={'jobs': 2}, max_builds=1),
 
         # test only: Fedora 28 x86_64, Intel Xeon E5-2630v3, 32 HT cores, 64GB RAM

zorg/trunk/buildbot/osuosl/master/config/status.py

return [
             fromaddr = "[e-mail]",
             sendToInterestedUsers = False,
             extraRecipients = ["[e-mail]"],
             subject="Build %(builder)s Failure",
             mode = "failing",
             builders = ["lldb-x64-windows-ninja"],
             addLogs=False,
             num_lines = 15),
+        InformativeMailNotifier(
+            fromaddr = "[e-mail]",
+            sendToInterestedUsers= False,
+            extraRecipients = ["[e-mail]"],
+            subject="Build %(builder)s Failure",
+            mode = "failing",
+            builders = ["fuchsia-x86_64-linux"]
+            addLogs=False),
         ]

zorg/trunk/zorg/buildbot/builders/FuchsiaBuilder.py

+import platform
+
+from buildbot.process.properties import WithProperties
+from buildbot.steps.shell import ShellCommand
+from buildbot.steps.source import SVN
+
+from zorg.buildbot.commands.CmakeCommand import CmakeCommand
+from zorg.buildbot.commands.NinjaCommand import NinjaCommand
+from zorg.buildbot.conditions.FileConditions import FileExists, FileDoesNotExist
+from zorg.buildbot.process.factory import LLVMBuildFactory, svn_repos
+
+def getToolchainBuildFactory(
+        clean=False,
+        test=True,
+        env=None, # Environmental variables for all steps.
+        extra_configure_args=None):
+    # Prepare environmental variables. Set here all env we want everywhere.
+    merged_env = {
+        "TERM" : "dumb", # Make sure Clang doesn't use color escape sequences.
+    }
+    if env is not None:
+        # Overwrite pre-set items with the given ones, so user can set anything.
+        merged_env.update(env)
+
+    src_dir = "llvm.src"
+    obj_dir = "llvm.obj"
+    install_dir = "llvm.install"
+
+    f = LLVMBuildFactory(
+            depends_on_projects=[
+                "llvm",
+                "clang",
+                "clang-tools-extra",
+                "compiler-rt",
+                "libcxx",
+                "libcxxabi",
+                "libunwind",
+                "lld"
+            ])
+
+    # Get Fuchsia SDK.
+    sdk_dir = "fuchsia.sdk"
+    sdk_platform = {
+        "Linux": "linux-amd64",
+        "Darwin": "mac-amd64",
+    }[platform.system()]
+    sdk_version = "latest"
+
+    sdk_url = WithProperties(
+        "https://chrome-infra-packages.appspot.com/dl/"
+        "fuchsia/sdk/%(sdk_platform)s/+/%(sdk_version)s",
+        sdk_platform=lambda _: sdk_platform,
+        sdk_version=lambda _: sdk_version)
+
+    f.addStep(RemoveDirectory(name="clean-sdk",
+                              dir=sdk_dir,
+                              haltOnFailure=True,
+                              description=["remove", "fuchsia sdk"],
+                              doStepIf=FileExists(sdk_dir)))
+
+    f.addStep(ShellCommand(name="fetch-sdk",
+                           command=["curl", "-SLf", "-o", "sdk.cipd", sdk_url],
+                           description=["download", "fuchsia sdk"],
+                           workdir=sdk_dir))
+
+    f.addStep(ShellCommand(name="extract-sdk",
+                           command=["unzip", "sdk.cipd"]
+                           description=["extract", "fuchsia sdk"],
+                           workdir=sdk_dir))
+
+    cleanCheckoutRequested = lambda step: step.build.getProperty("clean", default=False) or clean
+
+    # Clean up llvm sources.
+    f.addStep(RemoveDirectory(name="clean-llvm.src",
+                              dir=src_dir,
+                              haltOnFailure=True,
+                              description=["rm", "src", "dir"],
+                              doStepIf=cleanCheckoutRequested))
+
+    # Get sources.
+    for project in f.depends_on_projects:
+        _, baseURL = svn_repos[project]
+        f.addStep(SVN(name="svn-%s" % project,
+                      workdir=src_dir + "/" + project,
+                      baseURL=baseURL))
+
+    cleanBuildRequested = lambda step: step.build.getProperty("clean", default=step.build.getProperty("clean_obj")) or clean
+
+    # Clean up llvm build.
+    f.addStep(RemoveDirectory(name="clean-llvm.obj",
+                              dir=obj_dir,
+                              haltOnFailure=True,
+                              description=["rm", "build", "dir"],
+                              doStepIf=cleanBuildRequested))
+
+    # Configure.
+    if extra_configure_args is None:
+        cmake_options = []
+    else:
+        cmake_options = extra_configure_args[:]
+
+    # Some options are required for this stage no matter what.
+    CmakeCommand.applyRequiredOptions(cmake_options, [
+        ("-G",                      "Ninja"),
+        ("-DLLVM_ENABLE_PROJECTS=", "clang;lld"),
+        ("-DLLVM_ENABLE_RUNTIMES=", "compiler-rt;libcxx;libcxxabi;libunwind"),
+        ])
+
+    # Set proper defaults.
+    CmakeCommand.applyDefaultOptions(cmake_options, [
+        ("-DBOOTSTRAP_LLVM_ENABLE_LTO=", "OFF"),
+        ("-DLLVM_ENABLE_LTO=",           "OFF"),
+        ])
+
+    cmake_options.append(
+        WithProperties(
+            "-DCMAKE_INSTALL_PREFIX=%(workdir)s/" + install_dir,
+            "-DFUCHSIA_SDK=%(workdir)s/" + sdk_dir,
+        ))
+
+    CmakeCommand.applyRequiredOptions(cmake_options, [
+        ("-C", "../" + src_dir + "clang/cmake/caches/Fuchsia.cmake"),
+        ])
+
+    f.addStep(CmakeCommand(name="cmake-configure",
+                           options=cmake_options,
+                           path='../' + src_dir,
+                           haltOnFailure=True,
+                           description=["configure"],
+                           workdir=obj_dir,
+                           env=merged_env,
+                           doStepIf=FileDoesNotExist("CMakeCache.txt")))
+
+    # Build distribution.
+    f.addStep(NinjaCommand(name="ninja-build",
+                           targets=["stage2-distribution"],
+                           haltOnFailure=True,
+                           description=["build"],
+                           workdir=obj_dir,
+                           env=merged_env))
+
+    # Test llvm, clang and lld.
+    f.addStep(NinjaCommand(name="check",
+                           targets=["stage2-check-%s" % p for p in ("llvm", "clang", "lld")],
+                           haltOnFailure=True,
+                           description=["check"],
+                           workdir=obj_dir,
+                           env=merged_env
+                           doStepIf=test))
+
+    # Install distribution.
+    f.addStep(NinjaCommand(name="install",
+                           targets=["stage2-install-distribution"],
+                           haltOnFailure=True,
+                           description=["install"],
+                           workdir=obj_dir,
+                           env=merged_env))
+
+    return f